JP6903595B2 - Data analysis support system and data analysis support method - Google Patents

Description

The present invention relates to a technique for supporting data analysis using an information processing device.

With the progress of IoT (Internet of Things) technology and big data technology, there is an increasing need for data utilization across multiple business systems and sensors. Correlation rule mining (basket analysis / correlation analysis) is a data analysis algorithm for a large amount of data possessed by various business systems. Correlation rule mining is a technique that focuses on finding the correlation (correlation rule) of events that occur frequently and simultaneously from the target data, and is also applicable to non-numerical data. Correlation rules are expressed in the form of, for example, "event A and event B are likely to occur at the same time" and "product C is likely to be purchased at the same time as product D", and are expressed not only in data analysis but also in data search systems and information recommendation systems. Used.

As an index showing the strength of correlation, there are support level (occurrence rate in all samples), certainty level (combination frequency of premise part / conclusion part), and lift (degree of occurrence in combination or alone). Useful rules in analysis are extracted by performing threshold processing on the index. However, for example, if a rule with a low approval rating (small cases) is left, a large number of rules remain, and it becomes difficult to find a useful rule from these rules. On the other hand, if a rule with a high degree of support (many cases) is left, a self-evident (natural) rule remains as an event, and it becomes difficult to find a useful rule for business improvement and cause analysis.

In addition to the technology related to narrowing down a huge amount of rules extracted by such correlation rule mining, a means for generating a plurality of correlation rules indicating relationships between contents by targeting content consumption (product purchase, etc.) data, and the content consumption data. A means for calculating the rarity of each correlation rule using the above, and a means for narrowing down the correlation rule and generating a recommendation rule using the rarity. The total number of contents in the condition part and the consequent part of the correlation rule acquired from the content consumption data, the number of users corresponding to the correlation rule, and the number of contents consumed by each user corresponding to the correlation rule, were consumed. A recommendation rule generator (see Patent Document 1), which is characterized by calculating the rarity using the total number of contents and the number of users who have consumed each content in the condition part and the consequent part of the correlation rule, has been proposed. Has been done.

Japanese Unexamined Patent Publication No. 2014-222398

Rakesh Agrawal and Ramakrishnan Srikant, "Fast algorithms for mining association rules", Proceedings of the 20th International Conference on Very Large Data Bases, pp.487-499, 1994

At the initial stage of analysis work by matching data from multiple business systems, the purpose of analysis is clear, but the data to be analyzed is often not clear. That is, it is often unclear which data should be used to obtain data analysis results for verification and evaluation of analysis purposes.

In such a case, the analyst first matches the data that can be analyzed based on the data structure specifications such as the ER (Entity-Relationship) diagram in the original business system of the data to be analyzed, that is, In many cases, the approach is to first combine data tables that are close to each other from the viewpoint of the data structure of the original business system. This is to first select a combination of data in which the analysis result is likely to be obtained in order to obtain a useful analysis result within the limited analysis work man-hours.

In data analysis of multiple business systems, the above approach is often taken, so it is far from the viewpoint of the data structure of the original business system, especially between data tables that straddle business systems or even one business system. In many cases, it is not possible to sufficiently carry out analysis using data tables that are located at a distance. In addition, the approach of combining data tables that are close to each other from the viewpoint of the data structure of the original business system is often a rare (common) combination of data to be analyzed, and the analysis result also shows. Obvious (obvious) results are often obtained, which may not lead to useful results for business improvement or event cause investigation.

From the above, the correlation of data that is an unexpected combination of data to be analyzed, such as "data tables that straddle business systems" and "data tables that are far apart from the viewpoint of the data structure of the original business system", is particularly useful. Analysis results can be obtained.

However, the rarity obtained in the rarity calculation, which is a feature of the prior art, indicates the probability that the correlation rule will occur, and the above-mentioned "data tables straddling the business system" and "viewpoint of the data structure of the original business system" No consideration is given to unexpected data combinations such as "data tables that are far apart from each other". Therefore, in the prior art, it is not possible to narrow down the correlation rules including unexpected combinations as the combination of attributes included in the premise part and the conclusion part of the rule from the huge number of correlation rules extracted, and present a useful rule for the analyst. Can not.

One aspect of the present invention is a data analysis support system. This system stores the data table to be analyzed including multiple data tables, analyzes the storage device and the data table to be analyzed, and extracts multiple correlation rules showing the correlation of the attributes contained in the data table. A combination of attributes of the premise part and the conclusion part of the correlation rule is generated for each of the data relation model generation part and the correlation rule, which generates the data relation model showing the relationship between the extraction part and the plurality of data tables. It is provided with an unexpected degree calculation unit that obtains a distance between the attributes in the data relational model for each combination and calculates an unexpected degree based on the distance.

Another aspect of the present invention is a data analysis support system method performed by an information processing device including an input device, an output device, a storage device, and a processing device. In this method, a first step of preparing an analysis target data table including a plurality of data tables in a storage device, and a second step of generating a data relational model showing the relationship between the plurality of data tables. , Analyzing the data table to be analyzed and extracting multiple correlation rules showing the correlation of the attributes contained in the data table, the third step, and for each correlation rule, the attributes of the premise part and the conclusion part of the correlation rule It comprises a fourth step of generating a combination, finding the distance between the attributes in the data relational model for each combination, and calculating the degree of surprise based on the distance.

Unexpected rules can be narrowed down from a huge number of correlation rules, and useful information for business improvement and cause analysis can be quickly grasped.

It is a block diagram which shows the configuration example of a data analysis support system. It is a table figure which shows the format example of the analysis target data which is stored in the analysis target data storage part. It is a conceptual diagram which shows the format example of the entity table and the relation table of the data relation model storage part, and the relation generation principle. It is a table figure which shows the data format example of the correlation rule storage table of the correlation rule storage part. It is a top view which shows the screen example which the analysis performer takes in the analysis target data, calculates the correlation rule, and narrows down the correlation rule. It is a block diagram which shows the hardware configuration example of a data analysis support system. It is a flowchart which shows a series of steps of generating a data relational model in a data analysis support system, extracting a correlation rule, and calculating an unexpected degree. It is a flowchart which shows the detail of the procedure which the data relational model generation part generates the data relational model from the analysis target data table. It is a flowchart which shows the detail of the procedure which the data combination part joins the analysis target data table into one data table. It is a flowchart which shows the detail of the procedure which the surprise degree calculation unit calculates the surprise degree for each correlation rule based on a data relational model.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, the present invention is not construed as being limited to the description of the embodiments shown below. It is easily understood by those skilled in the art that a specific configuration thereof can be changed without departing from the idea or gist of the present invention.

In the configuration of the invention described below, the same reference numerals may be used in common among different drawings for the same parts or parts having similar functions, and duplicate description may be omitted.

When there are a plurality of elements having the same or similar functions, they may be described by adding different subscripts to the same code. However, if it is not necessary to distinguish between a plurality of elements, the subscript may be omitted for explanation.

The notations such as "first", "second", and "third" in the present specification and the like are attached to identify the components, and do not necessarily limit the number, order, or contents thereof. is not it. Further, the numbers for identifying the components are used for each context, and the numbers used in one context do not always indicate the same composition in the other contexts. Further, it does not prevent the component identified by a certain number from having the function of the component identified by another number.

The position, size, shape, range, etc. of each configuration shown in the drawings and the like may not represent the actual position, size, shape, range, etc. in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings and the like.

Components represented in the singular form herein shall include the plural form unless explicitly stated in the context.

An example of the example described below is data composed of a correlation rule extraction unit that analyzes an analysis target data table and extracts a plurality of correlation rules, and an entity table and a relational table that show the relationship between the analysis target data tables. Data relation for each correlation rule using the data relation model generation part that generates the relation model and the correlation rule extracted by the data relation model and the correlation rule extraction means for each combination of the attributes of the premise part and the conclusion part of the correlation rule. It is a data analysis support system characterized by including an unexpected degree calculation unit that calculates an unexpected degree based on the distance between entities in a model and the strength of a relation.

FIG. 1 is a diagram showing a configuration example of the data analysis support system 100 in this embodiment. As shown in FIG. 1, the data analysis support system 100 targeted by this embodiment is communicably connected to the user terminal 111. The data analysis support system 100 can be configured by, for example, a server, and the user terminal 111 can be configured by, for example, a personal computer. Both can be connected via a network, for example.

The data analysis support system 100 according to this embodiment has the analysis target data storage unit 101, the data relation model storage unit 102, the correlation rule storage unit 103, the data acquisition unit 104, and the data relation model generation unit 105 as functional components. A data combination unit 106, a correlation rule extraction unit 107, an unexpected degree calculation unit 108, a rule recommendation unit 109, and a user interface unit 110 are provided.

The data acquisition unit 104 receives the data acquisition request made by the analysis performer 112 to the user terminal 111, and acquires the analysis target data table stored in the analysis target data storage unit 101.

FIG. 2 shows an example of an analysis target data table stored in the analysis target data storage unit 101. In the example of FIG. 2, examples of the train data table 1011 and the station data table 1012 are shown as the data tables to be analyzed. Each table has column names 10111 and 10121, and stores predetermined numerical values or text data in each column. The data table has, for example, the data format shown in FIG. 2, and targets a data table having a general tabular data structure.

In the implementation of this embodiment, it is premised that the data to be analyzed is tabular data or data having a function equivalent to this, and it can be applied regardless of the type of industry or field. In this embodiment, data of various business systems in the railway field will be described as an example. As data examples of various business systems in the railway field, two tables, a train data table 1011 and a station data table 1012, are defined in the analysis target data storage unit. In each table, for example, identification information indicating an object that is a subject or an object, information on various physical quantities or status related to the object, and the like are stored.

The data relational model generation unit 105 generates a data relational model showing the relationship between the data tables to be analyzed, and stores the generated model in the data relational model storage unit 102. The data relational model stored in the data relational model storage unit 102 is a relation that defines the table name of the data table of the data relational model, the entity table that defines the column list of each table, and the relationship between the data tables of the data relational model. It consists of two tables.

FIG. 3 shows a conceptual diagram of the data relational model stored in the data relational model storage unit 102. As mentioned above, the data relational model includes the entity table 10210 and the relation table 10220.

The entity table 10210 is a list of column names of each data table stored in the analysis target data storage unit 101. The entity table 10210 has, for example, the data format shown in FIG. 3, and includes a table name 10211 and a corresponding column name 10212. The relation table 10220 includes a first table 10221, a column 10222 of the first table, a second table 10223, and a column 10224 of the second table.

In the example shown in FIG. 3, two tables, a train data table (1011 in FIG. 2) and a station data table (1012 in FIG. 2), are defined in the entity table 10210, and the train data table includes the enforcement date and the line section. , Train number, destination, first station, last station, total 6 columns, station data table 1012 defines train number, station name, other company's line entry, arrival time, departure time, delay time, total number of people staying 7 columns Will be done.

Further, in the relation table 10220, a relation between the train data table 1011 and the station data table 1012 is defined, and there is a relation between the train number column of the train data table 1011 and the train number column of the station data table 1012. Defined. Similarly, a relation is defined between the starting and ending station columns of the train data table 1011 and the station name column of the station data table 1012.

The data joining unit 106 generates one data table by joining in the horizontal direction using the column of the analysis target data table stored in the analysis target data storage unit as a key.

The correlation rule extraction unit 107 stores the correlation rule generated by performing the correlation rule mining on the data table generated by the data combination unit 106 in the correlation rule storage unit 103. The extraction of the correlation rule can be realized by using a known algorithm such as an a priori algorithm (see Non-Patent Document 1).

Correlation rule mining is an analysis algorithm that focuses on finding frequently simultaneous events in the data to be analyzed. A combination of events that occur frequently at the same time in the data to be analyzed, such as the simultaneity and relationships seen in the occurrence of multiple events, is extracted as a rule, and this rule is called a correlation rule. For example, when the relationship that the event Y under a certain event X occurs is recognized, it is described as "X⇒Y", the left side of the arrow (⇒) is the premise part (event X), and the right side is the conclusion part. It is called (event Y) and indicates the probability that Y will occur when event X occurs.

As is well known, in correlation rule mining, there are a total of three indicators of the strength of correlation: support, reliability, and lift. Support is a percentage of all data that includes an event. The degree of certainty is the rate at which the event of the conclusion part occurs after the event of the premise part occurs, and represents the strength of the relationship between the event of the premise part and the event included in the conclusion part. The lift is the ratio (confidence) of the event of the conclusion part occurring under the event of the premise part divided by the ratio of the event of the conclusion part occurring in all the data, and the event of the conclusion part is independent. It shows how much the event of the conclusion part occurs under the event of the premise part more than the rate of the event of the conclusion part.

For example, "The rate at which event Y occurs under event X is 60%, the rate at which event X and event Y occur at the same time is 20% in all data, and the rate at which event Y occurs under event X. The correlation rule "X⇒Y" that "is 2.5 times the rate at which event Y occurs alone in all data" is shown as support = 20%, certainty 60%, and lift 2.5. .. There may be a plurality of events included in the premise part and the conclusion part. In addition, the "event" included in the premise part and the conclusion part may be referred to as an "item" or an "attribute". In the following description, it will be referred to as an "attribute" rather than an "event".

FIG. 4 shows an example of a data format of the correlation rule storage table 1030 included in the correlation rule storage unit 103. The correlation rule storage table 1030 includes a premise part 1031, a conclusion part 1032, a support degree 1033, a certainty degree 1034, a lift 1035, and an unexpected degree 1036 as data items. In the example of FIG. 4, the correlation rule "train number (T100) ⇒ vehicle ID (M1-01)" has a support level of 8.30%, a certainty level of 60%, and a lift of 2.3. In this example, not only the data values (T100 and M1-01 in the above correlation rule) but also the information about which column of which table the data value belonged to (the train in the train data table 1011 in the above correlation rule). The number and vehicle data table (vehicle ID) are also stored in the premise part and the conclusion part, but the data table is omitted. The "surprise degree" will be described later in the description of the surprise degree calculation unit 108.

The unexpected degree calculation unit 108 compares the events included in the premise part and the conclusion part of the correlation rule with the data relation model generated by the data relation model generation unit 105 for each correlation rule extracted by the correlation rule extraction unit 107. The degree of surprise is calculated and stored in the correlation rule storage unit 103. The calculated unexpectedness is stored in the unexpectedness column (FIG. 4) of the correlation rule storage table 1030 of the correlation rule storage unit 103.

The rule recommendation unit 109 receives a correlation rule narrowing request from the analyst, a support degree defined by the analyst performer, a certainty degree, a lift, and a total of four threshold values for the unexpected degree, and receives the correlation rule storage unit 103. By performing threshold processing on all the stored correlation rules, the correlation rules are narrowed down, and the narrowed down result is returned to the user terminal 111. The threshold processing leaves rules having a value higher than the threshold set for each index, and removes rules having a value equal to or lower than the threshold. A rule having a value higher than the threshold value is left for each of the four indicators of support, certainty, lift, and unexpectedness.

The user interface unit 110 generates a screen 1101 in which the analyst executes the analysis target data, calculates the correlation rule, and narrows down the correlation rule.

FIG. 5 shows a plan view of an example of the screen generated by the user interface unit 110. As shown in FIG. 5, an example of this screen is composed of a header unit 1102, a threshold value setting unit 1103, a correlation rule list display unit 1104, and a data relational model display unit 1105. In the header section 1102, a data import button for the analyst to import the analysis target data, a correlation rule calculation button for extracting the correlation rule for the analysis target data and calculating the degree of surprise, and a threshold setting of the extracted correlation rule are set. It is composed of a correlation rule narrowing down button for narrowing down with the threshold value set in the part 1103.

When the analyst 112 presses the data acquisition button, the data acquisition request is transmitted from the user terminal 111 to the data acquisition unit 104. When data is taken in from the analysis target data storage unit 101, the data relational model generation unit 105 generates a data relational model, and the generation result is displayed on the data relational model display unit 1105 as, for example, an ER diagram. By using the entity addition / edit button, relation addition button, and deletion button for the generated model, the analyst may adjust / change the model according to the purpose of analysis and the knowledge of the analyst. .. Further, the data acquisition is not limited to all the data tables of the analysis target data storage unit 101, and the analysis practitioner 112 may select an arbitrary data table. In this case, the data table name selected by the analyst along with the data acquisition request is also transmitted to the data acquisition unit 104.

When the analyst 112 presses the correlation rule calculation button, the correlation rule extraction unit 107 extracts the correlation rule from the data table generated by the data combination unit 106, and the unexpected degree calculation unit 108 extracts each correlation. Calculate the degree of surprise for the rule based on the data-related model. When the calculation of the degree of surprise for all the rules is completed, all the correlation rules are listed in the correlation rule list display unit 1104.

When the analyst 112 presses the correlation rule narrowing down button, the support, certainty, lift, threshold and rule recommendation request set for the threshold setting unit 1103 are sent to the rule recommendation unit 109. Will be sent. The result of narrowing down the rules by the rule recommendation unit 109 is displayed on the screen 1101.

In the example of FIG. 5, the threshold values are set to support degree = 3.0%, certainty degree = 20.0%, lift = 1.5, and unexpected degree = 80.0%. As a result, among the extracted correlation rules, the rule narrows down the values higher than the threshold value for each index, and the rule is displayed on the correlation rule list display unit 1104. In the example of FIG. 5, the correlation rule "train number (T102) ⇒ gradient (0.5-1.0%)" has support = 7.5%, certainty = 50%, lift = 2.6, unexpectedness = It is 100%, and each index remains as a rule having a value higher than the threshold value set by the threshold value setting unit 1103. Details of these functional components will be described later using a flowchart.

FIG. 6 is a diagram showing a hardware configuration example of the data analysis support system 100. The data analysis support system 100 includes a CPU (central processing unit) 201, an HDD (magnetic disk device) 202, a memory 203, an input unit 204, a display unit 205, and a communication unit 206. The CPU 201 executes data input / output, reading, storage, and various processes. The HDD 202 is a device for storing data, and the memory 203 is a device for temporarily storing programs and data. Both are collectively called a storage device. The input unit 204 is an input device that receives an operation input from the user. The display unit 205 is a device that displays data to the user and is one of the output devices. The communication unit 206 is a device that communicates with the user terminal 111 and transmits / receives data. Each of these devices can be realized as each configuration of a general computer.

The analysis target data storage unit 101, the data relational model storage unit 102, and the correlation rule storage unit 103 of FIG. 1 are realized by, for example, the HDD 202. Each part of the data acquisition unit 104, the data relation model generation unit 105, the data combination unit 106, the correlation rule extraction unit 107, the unexpectedness calculation unit 108, and the rule recommendation unit 109 of the first embodiment is stored in, for example, the memory 203. The program is executed by the CPU 201, which is executed by controlling the hardware such as the CPU 201, the HDD 202, the memory 203, the input unit 204, the display unit 205, and the communication unit 206.

The configuration of the above data analysis support system 100 may be configured by a single computer, or any part of the CPU 201, HDD 202, memory 203, input unit 204, and display unit 205 may be networked via the communication unit 206. It may be composed of other computers connected by. Further, in this embodiment, a function equivalent to the function configured by software can be realized by hardware such as FPGA (Field Programmable Gate Array) and ASIC (Application Specific Integrated Circuit).

FIG. 7 is an overall flowchart showing a series of procedures for generating a data relational model, extracting a correlation rule, and calculating an unexpected degree in the data analysis support system 100.

The data acquisition unit 104 receives the analysis target data acquisition request input on the screen of FIG. 5 displayed on the user terminal 111 by the analysis performer 112, and acquires the analysis target data table from the analysis target data storage unit 101. To do. Then, the data relational model generation unit 105 generates a data relational model for the acquired data table (S301).

The data joining unit 106 generates one data table by internally joining the data table to be analyzed in the horizontal direction using a time-series data item as a key (S302).

The correlation rule extraction unit 107 receives the correlation rule extraction request input by the analyst 112 on the screen of FIG. 5 displayed on the user terminal 111, and extracts the correlation rule (S303).

The unexpected degree calculation unit 108 calculates the unexpected degree for each correlation rule extracted by the correlation rule extraction unit 107 by comparing with the data relational model generated by the data relational model generation unit 105 (S304).

The rule recommendation unit 109 receives the correlation rule narrowing request input by the analyst on the screen of FIG. 5 displayed on the user terminal 111, and the threshold values for the degree of support, the degree of certainty, the lift, and the degree of unexpectedness, and correlates them. Threshold processing is performed on the support, certainty, lift, and unexpectedness calculated for each rule to narrow down the rules, and the result is returned to the user terminal 111 (S305).

The data relational model generation S301 may be performed after the correlation rule extraction S303. Alternatively, it may be created and saved in advance before the processing of FIG. 7.

Details of the processing procedure performed by the data-related model generation unit 105 will be described later in the flowchart shown in FIG. Details of the processing procedure performed by the data combining unit 106 will be described later in the flowchart shown in FIG. The details of the processing procedure performed by the unexpected degree calculation unit 108 will be described later in the flowchart shown in FIG.

FIG. 8 is a flowchart showing the details of the procedure S301 in which the data relational model generation unit 105 generates the data relational model from the analysis target data table.

The data relational model generation unit 105 acquires a list of column names of each data table for all the analysis target data tables acquired by the data acquisition unit and stores them in the entity table 10210 (see FIG. 3) of the data relational model (S3011). ).

Loop processing is performed for the number of selection methods for selecting two tables from all the acquired data tables (S3012).

Subsequently, the two tables selected in S3012 are looped by the number of columns multiplied by the number of columns in each table (S3013). This process is synonymous with fixing the columns of one table and performing the process for all the columns of the other table for the fixed columns.

Of the two tables selected in S3012, the names of the columns defined in one table and the columns defined in the other table are compared (S3014).

It is determined whether the names of the columns to be compared are partially matched or exactly matched (S3015).

If there is a partial match, it is determined that there is a relation between the two columns, and the relation is stored in the relation table 10220 of the data relation model storage unit 102 (S3016).

A description of the data relational model generation process when the data to be analyzed is the train data table 1011 and the station data table 1012 shown in FIG. 2 is shown. The table names of the train data table 1011 and the station data table 1012 and the column names of each table are acquired, and the acquisition results are stored in the entity table 10210 shown in FIG.

Then, the selection method of selecting two tables from all the acquired data tables is calculated. In this example, there are two target data tables, and there is only one way to select two tables from them, so loop processing is performed only once.

Subsequently, the number of columns in each of the two selected tables is 6 × 7 = 42 because the train data table 1011 is 6 and the station data table 1012 is 7, and loop processing is performed 42 times. First, it is determined whether or not the execution date column of the train data table 1011 and all the columns of the station data table 1012 and the character strings partially match (the total number of loops is 7). Similarly, for the remaining 5 columns of the train data table 1011 as well, it is determined whether or not the character strings of all the columns of the station data table 1012 partially match.

In this example, since the train number column of the train data table 1011 partially matches the train number column of the station data table 1012, it is determined that there is a relation 30001 between the columns, and the determination result is shown in FIG. Store in table 10220. Further, regarding the starting station and ending station columns of the train data table 1011 as well, since the character strings partially match the station name column of the station data table 1012, it is determined that there is a relation 30002 between each column, and the result is shown in the relation table 10220. To store.

By using the data stored in the entity table 10210 and the relation table 10220 generated by the data relational model generation process, the data relational model is represented in the form of a schematic class diagram as shown in the data relational model display unit 1105 of FIG. be able to.

The train data table 1011 and the station data table 1012 shown in FIG. 2 are represented as a train class and a station class, respectively, and the relationship between the two data tables is represented by a line connecting the train class and the station class. In the example of the data relational model shown in FIG. 5, although the description is omitted in FIG. 2, the classes such as vehicles and railroad tracks are also displayed. In this example, in order to improve visibility, the relationship between the vehicle entity and the ground equipment entity (the vehicle class speed log and indoor temperature log column partially match the ground equipment class operation log and alarm log column, so the relationship Some relations such as (determined to be present) are omitted.

In addition, the relation in the data relation model generation process defines not only the relation between the structures of the data table to be analyzed, but also the hierarchical relation between the structures seen unique to the industry, and the proximity and front-back relations on the position and the route. You may. For example, in the field of railways, a hierarchical relationship between structures such as train-vehicle-vehicle parts such as a vehicle that composes a train and various vehicle parts that compose a vehicle can be considered. By predefining the hierarchical relationship of, it is possible to define the events that occur in the same structure. In addition, as the proximity and front-to-back relationship on the position and route, by predefining the information on the order of stations and the lines running in parallel, the relationship of events that spread between adjacent stations and the transfer route and structure You can define the relationship that spreads to the train.

FIG. 9 is a flowchart showing the details of the procedure S302 in which the data joining unit 106 joins the data table to be analyzed into one data table.

The data combining unit 106 performs loop processing on all the analysis target data tables acquired by the data acquisition unit 104 (S3021).

For each column of the table, the data value defined in the column is acquired and the data type is determined (S3022).

It is determined whether or not the data type determined in S3022 corresponds to any of the time stamp type, the date type, and the time type (S3023).

If the column corresponds to any of the time stamp type, the date type, and the time type, the column is determined to be a column indicating a time series (S3024).

After the data type is determined for all the data tables to be analyzed and the column determination of the time series is completed, the data table is moved horizontally between the same data types using the column determined to indicate the time series as a key. By internally joining, the data to be analyzed becomes one data table (S3025).

In the above example, the data type is determined by analyzing the data value in column units in S3022, but if it is decided in which column to join the data tables, which column indicates the series in advance. The data type may be determined by preparing a defined user-defined data table in advance and referring to this user-defined table. For example, if there is a column called kilometer that indicates the position of the inspection result on the track, as in the track inspection measurement log result data table, the kilometer is not a time series but a position series data. You may want to join the data tables as a key. In this case, the kilometer is defined in the user-defined table in advance, and it is determined by referring to this user-defined table whether or not there is a column containing the kilometer for all the data tables to be analyzed, and the kilometer is determined. Horizontally join data tables using columns as keys.

In addition, the minimum unit of the data value of the column that is the join key of the data table and the timing of data acquisition may differ for each data table. For example, in a column showing the time in one table, data is acquired in units of 30 seconds, while in another table, data is acquired in units of 1 minute. The minimum unit may be different. Further, even in the same data table in units of 30 seconds, since the data acquisition timings are different, the base time may be "10:00:05" and "10:00:12". In such a case, the analysis target data table may be subjected to preprocessing such as aligning the minimum unit or aligning the data value in the column indicating the time to a coarser unit according to the request of the analyst. ..

FIG. 10 is a flowchart showing the details of the procedure S304 in which the unexpected degree calculation unit 108 calculates the unexpected degree for each correlation rule based on the data relational model.

After the processing of the correlation rule extraction unit 107 is completed, the unexpected degree calculation unit 108 performs loop processing for the number of extracted correlation rules (S3041).

Acquire a list of attributes included in the premise part and the conclusion part for the correlation rule to be looped (S3042). As already mentioned, an attribute refers to an event included in the premise part and the conclusion part.

How to select 2 attributes from the acquired attribute list Perform loop processing for the number of selections (S3043).

The distance in the data relational model between the selected two attributes is calculated (S3044). The distance between two attributes in the data relational model is the distance between the classes to which the attribute belongs. The distance between the classes can be grasped as the number of relationships connecting the classes in the data relational model shown in FIG. 5, for example. For example, the distance between the train class and the track class is 2. Therefore, the distance between the train class attribute "enforcement date" and the track class attribute "km" is 2.

Note that what is generally called an entity or table in the data model is called a class or object in the object model. In this specification, the terms of entities, tables, and classes may be replaced and understood.

After the loop processing of S3043 is completed, in (the sum of the distances in the data relational model for all the selection methods for selecting 2 attributes from all the attributes included in the premise part and the conclusion part), (the distance between the 2 attributes in the selection method for selecting 2 attributes) The unexpectedness is calculated by dividing the sum of the distances of those having 2 or more, and this is stored in the unexpectedness column of the rule in the correlation rule storage table 1030 (S3045).

Here, based on the data relational model shown in the data relational model display unit 1105 of FIG. 5, the correlation rule 1 "train number (T102) ⇒ gradient (0.5-1.0%)" and the correlation rule 2 "train number (train number (0.5-1.0%)" An example of calculating the degree of surprise for each of "T200) ⇒ alarm log (A200) and room temperature log (26.0-26.5 ° C.)" will be described.

Correlation rule 1 "Train number (T102)" and "Gradient (0.5-1.0%)" as a list of attributes included in "Train number (T102) ⇒ Gradient (0.5-1.0%)" 2 attributes are acquired. Each attribute is a train number column of the train data table 1011 and a gradient column of the track data table, respectively. Since there is only one way to select 2 attributes from the total of 2 attributes included in the premise part and the conclusion part, the loop processing is performed only once. When calculating the distances in the data relational model for these two attributes, "train number (T102)" belongs to the train class and "gradient (0.5-1.0%)" belongs to the track class. Class is a distance of 2 across the vehicle class. When calculating the degree of surprise, (the sum of the distances in the data relational model for all the selection methods for selecting 2 attributes from all the attributes included in the premise part and the conclusion part) is 2, (the distance between 2 attributes among the selection methods for selecting 2 attributes). Is 2 or more, but the total distance) is also 2, 2/2 = 1, and the degree of surprise is 1 (100%).

In addition, as a list of attributes included in correlation rule 2 "train number (T200) ⇒ alarm log (A200), indoor temperature log (26.0-26.5 ° C)", "train number (T200)" and "alarm" Three attributes of "log (A200)" and "room temperature log (26.0-26.5 ° C.)" are acquired. Each attribute is a train number column of the train data table 1011, an alarm log column of the ground equipment data table, and an indoor temperature log column of the vehicle data table. How to select 2 attributes from a total of 3 attributes included in the premise part and conclusion part is "Train number (T200) and alarm log (A200)", "Train number (T200) and room temperature log (26.0-26.). Since there are a total of three types of "5 ° C.)" and "alarm log (A200) and room temperature log (26.0-26.5 ° C.)", the loop processing is performed only three times. When calculating the distance in the data relation model for each combination of attributes, the distance between "train number (T200) and alarm log (A200)" = distance between train class and ground equipment class = 3, "train number (T200)" ) And the indoor temperature log (26.0-26.5 ° C) = distance between the train class and the vehicle class = 1, "alarm log (A200) and indoor temperature log (26.0-26.5 ° C)" ) ”= Distance between vehicle class and ground equipment class = 2. Therefore, when calculating the degree of surprise, (the sum of the distances in the data relational model for all the selection methods for selecting 2 attributes from all the attributes included in the premise part and the conclusion part) is 3 + 1 + 2 = 6, (2 of the selection methods for selecting 2 attributes). The sum of the distances between the attributes is 2 or more) = 3 + 2 = 5, 5/6 = 0.83, and the degree of surprise is 0.83 (83%).

By calculating the degree of surprise as described above, as each attribute of the premise part and the conclusion part of the correlation rule, the more the model has a distance of 2 or more, the larger the degree of surprise becomes. In other words, a rule that has a combination of attributes that deviates from the relationship between general objects and events is evaluated as an unexpected relationship. In this way, by introducing an evaluation index called unexpectedness, it is possible to quantitatively evaluate an unexpected combination of data from a huge number of correlation rules, and it is possible to effectively narrow down the rules.

In the above example, (the sum of the selection methods for selecting two attributes where the distance between the two attributes is 2 or more) is the numerator, but (the distance between the two attributes among the selection methods for selecting the two attributes is m or more). It is also possible to arbitrarily set the parameter m as, for example, 3 or more as the sum of the distances). The larger m is, the more unexpected the rule is emphasized.

In the first embodiment, the degree of surprise is calculated by using only the result of the presence / absence of the relation between the data tables. In another example, the unexpected degree calculation unit 108 may consider the weight of the relation in the unexpected degree calculation. In the second embodiment, an example in which the weight of the relation is considered in the calculation of the degree of surprise is shown.

The relation weight can be defined by the number of pairs of columns determined to have a relation between the two tables. The weight of the relation can numerically express the strength of the relationship in the data structure between the two tables.

For example, in the example shown in FIG. 3, a total of 3 records (pairs) of relations are defined in the relation table 10220 between the train data table 1011 and the station data table 1012. Therefore, the weight between the train data table 1011 and the station data table 1012 is 3. It can be said that the data tables at both ends of the relation with a larger weight are more likely to be selected as a combination as the data to be analyzed in the data analysis. Therefore, the data tables at both ends of the relation with a large weight are not surprising as a combination of data, and are considered to be natural.

Therefore, in the unexpected degree calculation in consideration of the relation weight, in the unexpected degree calculation S304 of FIG. 7, the distance of the data relational model between the two tables is multiplied by the reciprocal of the relational weight, and the data relation is performed. The calculation is performed after correcting the distance in the model. By doing so, it is possible to calculate the degree of surprise in consideration of the strength of the relevance in the data structure.

In addition, depending on the combination of data to be analyzed, in the calculation of the distance between any two columns in the data relational model, there may be a plurality of routes connecting the two classes to which the two columns belong, or there may be a loop route. .. In such a case, in the calculation of the distance between two columns in the data model, the unexpectedness is calculated by acquiring the distance of the shortest route or setting a restriction that the route that has passed once does not pass twice. You may.

That is, in the first embodiment, the distance between the attributes is obtained by counting the number of relations existing between the data tables including the attributes of the premise part and the conclusion part of the correlation rule. In the second embodiment, the weight of the relation is calculated by the number of pairs whose column names partially match or exactly match between the two tables associated by the relation, and the reciprocal of the weight is the correction value of the relation between the two tables. Then, the distance between the attributes is obtained by adding the correction value of the relation existing between the data tables including the attributes of the premise part and the conclusion part of the correlation rule. In the first embodiment, the parameter m is basically a natural number, but in the second embodiment, the parameter m does not have to be a natural number because of the weighting process.

According to the examples described above, the analyst can narrow down the rules while judging the obvious and unexpected data combinations from the huge number of correlation rules, and can improve the business and analyze the cause. You can quickly grasp useful information for this purpose.

Data analysis support system 100, data storage unit 101 to be analyzed, data relation model storage unit 102, correlation rule storage unit 103, data acquisition unit 104, data relation model generation unit 105, data combination unit 106, correlation rule extraction unit 107, unexpected Degree calculation unit 108, rule recommendation unit 109, user interface unit 110

Claims (15)

It is a data analysis support system
A storage device that stores data tables to be analyzed, including multiple data tables,
A correlation rule extraction unit that analyzes the analysis target data table and extracts a plurality of correlation rules indicating the correlation of the attributes included in the data table.
A data relational model generation unit that generates a data relational model showing the relationship between the plurality of data tables, and
For each correlation rule, a combination of attributes of the premise part and the conclusion part of the correlation rule is generated, the distance between the attributes in the data relational model for each combination is obtained, and the degree of surprise is calculated based on the distance. , Surprising degree calculation part,
A data analysis support system characterized by being equipped with. The unexpected degree calculation unit is used for each of the correlation rules.
In "The sum of the distances in the data relational model for all the selection methods for selecting two attributes from all the attributes included in the premise part and the conclusion part of the correlation rule", "The distance between the two attributes in the selection method for selecting the two attributes is m. By dividing the "sum of the above distances"
The unexpectedness is calculated for each correlation rule.
The data analysis support system according to claim 1. The m is 2.
The data analysis support system according to claim 2. The unexpectedness calculation unit
Performs weighting on association between the data table, corrects the pre Ki距away by the weighting,
The data analysis support system according to claim 1. The data relational model is
It is composed of an entity table indicating the attribute names included in each of the data tables and a relation table indicating whether or not the attribute names included in each of the data tables are related.
The data analysis support system according to claim 1. The data analysis support system according to claim 1.
Provided with a user interface unit that generates a screen for an analyst for presenting the unexpectedness calculated for each correlation rule.
A data analysis support system featuring. The data analysis support system according to claim 6.
Provided with a rule recommendation unit that receives a predetermined threshold value for the unexpected degree and narrows down the correlation rule having a value higher than the received threshold value.
A data analysis support system featuring. An input device, an output device, storage device, and a data content析支援方method executed by the information processing apparatus including a processing device,
The first step of preparing an analysis target data table including a plurality of data tables in the storage device, and
The second step of generating a data relational model showing the relationship between the plurality of data tables, and
The third step of analyzing the analysis target data table and extracting a plurality of correlation rules indicating the correlation of the attributes contained in the data table, and
For each correlation rule, a combination of attributes of the premise part and the conclusion part of the correlation rule is generated, the distance between the attributes in the data relational model for each combination is obtained, and the degree of surprise is calculated based on the distance. , 4th step,
A data analysis support method characterized by being equipped with. The fourth step is for each correlation rule.
In "The sum of the distances in the data relational model for all the selection methods for selecting two attributes from all the attributes included in the premise part and the conclusion part of the correlation rule", "The distance between the two attributes in the selection method for selecting the two attributes is m. By dividing the "sum of the above distances"
The unexpectedness is calculated for each correlation rule.
The data analysis support method according to claim 8. The m is 2.
The data analysis support method according to claim 9. The fourth step is
Performs weighting on association between the data table, corrects the pre Ki距away by the weighting,
The data analysis support method according to claim 8. The data relational model is
It is composed of an entity table indicating the attribute names included in each of the data tables and a relation table indicating whether or not the attribute names included in each of the data tables are related.
The data analysis support method according to claim 8. A fifth step in which the output device displays a screen for inputting a threshold value for the unexpected degree, and
It has a sixth step of receiving the threshold value from the input device and narrowing down the correlation rule having a degree of surprise higher than the threshold value.
The data analysis support method according to claim 8. Each of the plurality of data tables includes a column name indicating an attribute name, and includes a column name indicating an attribute name.
The second step is
By associating two tables whose column names partially or completely match among the plurality of data tables by relation, a data relational model showing the relationship between the plurality of data tables is generated.
The fourth step is
The distance between the attributes is obtained by counting the number of the relations existing between the data tables including the attributes of the premise part and the conclusion part of the correlation rule.
The data analysis support method according to claim 8. The fourth step is
The weight of the relation is calculated by the number of pairs whose column names partially match or exactly match between the two tables associated with the relation.
The reciprocal of the weight is used as the correction value for the relationship between the two tables.
The distance between the attributes is obtained by adding the correction values of the relations existing between the data tables including the attributes of the premise part and the conclusion part of the correlation rule.
The data analysis support method according to claim 14.

Images