JP7227772B2 - DATA ASSET ANALYSIS SUPPORT SYSTEM AND DATA ANALYSIS METHOD - Google Patents

Description

The present invention relates to techniques for performing data asset analysis.

There are data analysis systems that respond to business issues, such as failure point diagnosis support systems and failure sign detection systems that use various data such as sensor data of machines and equipment and data recorded by people.

Patent document 1 is a technique for providing a data analysis system capable of flexibly adjusting the immediacy and safety of data analysis according to the application and purpose of the analysis. Patent Document 1 has a preprocessing program that preprocesses raw data from each store and an analysis program that analyzes the preprocessed data. Based on a destination list that associates each store with a preprocessing program, the data collection device preprocesses the data corresponding to each store, and attaches a store ID to the preprocessed preprocessed data. , to a center server, and the center server reads out an analysis program corresponding to the store ID and analyzes the preprocessed data.

JP 2010-277534 A

In Patent Document 1, the information (ID) of each store that generates raw data is associated with the preprocessing program and the analysis program, and the preprocessing program and analysis program that match the raw data of each store are selected. This has realized immediacy of data analysis and improved efficiency of data analysis.

However, in modern data asset analysis environments, data owners and analysts are completely separate, and analysts may not be informed of the type of data provided by data owners. In other words, there are cases where the data owner does not provide information that can specify the data type, such as store information, as in Patent Document 1.

In such an environment, it is necessary for people to understand the characteristics of the data used for analysis and select the appropriate analysis method. There was a problem that it was complicated to understand the characteristics of the data, and that it required a lot of man-hours and lead time to prepare for the actual analysis.

Therefore, the object of the present invention is to reduce the number of man-hours for pre-analysis of data that does not have data type information, etc., when the type of data to be analyzed is unknown. To provide a data asset analysis support system and method for greatly shortening the period.

An example of the data asset analysis support system of the present invention for solving the above problems is a data asset analysis support system having a processing unit, a storage device, and an output device, wherein the storage device includes inputs provided by the data owner. It stores data and an analysis template that stores feature amounts of analysis target data and analysis flows in association with each other. Then, the processing unit generates a feature amount of the input data stored in the storage device, selects an analysis flow for the input data based on the generated feature amount and the analysis template, and selects an analysis flow for the input data. and transmitting analysis results according to the analysis flow to the output device, and the output device is configured to output the analysis results according to the analysis flow.

According to the present invention, it is possible to reduce man-hours for understanding the characteristics of data and shorten the period until business problem solving. Also, an appropriate analysis flow can be selected according to data characteristics.

In addition, according to the present invention, results can be obtained easily, so it is possible to motivate people to accumulate data.

Problems, configurations, and effects other than those described above will be clarified by the following description of the mode for carrying out the invention.

1 is a hardware configuration diagram of a data asset analysis support system; FIG. It is a processing flow diagram of the data asset analysis support system. It is the figure which showed the whole data asset analysis support system outline|summary. FIG. 4 is a diagram showing an overview of template execution processing of the data asset analysis support system; It is a flow chart of feature quantity generation in the data asset analysis support system. It is a flow chart of feature quantity generation in the data asset analysis support system. FIG. 10 is a diagram showing an example of input data in which date and time items are rearranged in the first column; It is a template selection in a data asset analysis support system, and an execution flowchart. FIG. 10 is a diagram showing an analysis template table; FIG. It is a figure which shows an analysis data table. It is a figure which shows the output image of data analysis. It is a figure which shows the output image of data analysis. It is a figure which shows the output image of data analysis.

Embodiments of the present invention will be described below with reference to the drawings. The following description and drawings are examples for explaining the present invention, and are appropriately omitted and simplified for clarity of explanation. The present invention can also be implemented in various other forms. Unless otherwise specified, each component may be singular or plural.

In the following description, various types of information may be described using expressions such as “table”, but the various types of information may be expressed using data structures other than these. "XX table" and the like are sometimes called "XX information" to indicate that they do not depend on the data structure. When describing identification information, expressions such as “identification information”, “identifier”, “name”, “ID”, and “number” are used, but these can be replaced with each other.

When there are a plurality of constituent elements having the same or similar functions, they will basically be given the same reference numerals for explanation, but even if the functions are the same, the means for realizing the functions may differ.

Further, in the following description, there are cases where processing performed by executing a program will be described. Since processing is performed using storage resources (eg, memory) and/or interface devices (eg, communication ports), etc., processing may be performed by a processor.

A program may be installed on a device, such as a computer, from a program source. The program source may be, for example, a program distribution server or a computer-readable storage medium. When the program source is a program distribution server, the program distribution server may include a processor and storage resources for storing the distribution target program, and the processor of the program distribution server may distribute the distribution target program to other computers. Also, in the following description, two or more programs may be implemented as one program, and one program may be implemented as two or more programs.

An overview of representative embodiments of the present invention will be described. FIG. 1 is a hardware configuration diagram of a data asset analysis support system. As shown in FIG. 1, the data asset analysis support system 104 includes an interface 105 for communicating with external devices, an input/output device 130, a processing unit (Central Processing Unit) 106, a memory 108, an HDD, an SSD, etc. It is configured including a configured storage device 107 .

The interface 104 is connected via a network 103 to an edge computer 150 that collects data from various sensors and a data lake 151 that stores sensor data and the like. receive. Output devices of the input/output device 130 include output devices such as display devices and printers, and input devices include input devices such as mice, keyboards, and touch panels. The memory 108 is composed of a volatile memory such as a DRAM (Dynamic Random Access Memory).

A CPU 106 constituting a processing unit reads various programs 110 to 115 stored in a storage device 107 into a memory 108 and executes various programs, thereby realizing various functions corresponding to the programs. The memory 108 stores various programs to be executed by the CPU 106 and temporarily stores processing results of the CPU 106 .

Here, the various programs include a feature generation program 110, a template management program 111, a template selection program 112, a template execution program 113, an analysis result report output 114, and a visualization execution program 115. Various processes are realized by the CPU 106 executing these programs.

The storage device 107 also stores input data 120 input from the edge computer 150 and data lake 151 to be analyzed, analysis data 121 generated from the input data by the feature generation program 110, and managed by the template management program 111. and stores a plurality of templates 122 containing analysis flows that are selected by the template selection program 112 and perform analysis of input data. A template selection program 112 selects one template for analyzing input data from a plurality of templates.

FIG. 2 is a processing flow diagram of the data asset analysis support system 104. As shown in FIG. When the input data 120 to be analyzed is input from the edge computer 150 or the data lake 151, the feature amount generation program 110 generates feature amounts such as the average value, distribution, and deviation value of the input data 120 (A ).

One template is selected from a plurality of templates 122 managed by the template management program 111 based on the analysis data generated by the feature generation program 110 (B). The template contains an analysis flow for analyzing input data. Based on the selected template, the template execution program 113 executes the template (executes the analysis flow) (C), the analysis result report output program 114 transmits the analysis result to the output device, and the output device displays the analysis result ( E).

Here, the template corresponds to the analysis flow selected from the feature amount, and the template execution corresponds to the execution of the analysis flow selected from the feature amount.

The input data is input to the existing visualization execution program 115, the average value, distribution, deviation value, etc. of the input data 120 are visualized (D) and transmitted to the output device, and the output device outputs the visualized data display.

Although predictive diagnosis and classification diagnosis are shown as the analysis template 122 in FIG. 2, an analysis flow for failure diagnosis may be included.

FIG. 3 is a diagram showing an overall overview of the data asset analysis support system. The visualization system 115, the analysis template management system 111, and the like illustrated in FIG. 3 represent functions realized by the CPU 106 executing various programs. Each system is composed of a control section corresponding to the CPU 106 and a storage section corresponding to the storage device 107 and the memory 108 .

The input data 120 provided from the edge computer 150 and the data lake 151 are input to the visualization system 115 and feature value generation system 110, respectively. The analysis template management system is managed by the template management program 111, and stores an analysis template table 122 (FIG. 9A) that associates and manages data feature amounts and analysis flows. Information in the analysis template table may be registered in advance by an analyst.

Analysis results statistically processed by visualization system 115 are sent to output device 130 as analysis result report 114 . Images output on output device 130 are shown in FIGS.

On the other hand, the feature amount generation system 110 calculates the feature amounts such as the average value, distribution, and deviation value of the input data, and associates the feature amounts calculated for each input data file with the analysis data table (Fig. 9B). memorize. The template selection and execution systems 112 and 113 analyze the input data based on the analysis template table 122 that manages the feature values calculated by the feature value generation system 110 and the data feature values and analysis flows in association with each other. select and execute.

The template selection/execution systems 112 and 113 are implemented by the CPU 106 executing the template selection program and template execution program shown in FIG.

The analysis results of the template selection/execution systems 112 and 113 are also output as an analysis result report by an output device, similar to the output of the visualization system 125. FIG.

FIG. 4 is a diagram showing an outline of template execution processing of the data asset analysis support system. Each system shown in FIG. 4 represents functions realized by the CPU 106 executing various programs.

When the input data 120 is input to the feature amount generation system 110 (A), the feature amount such as the average value, distribution, and deviation value of the input data is generated by the feature amount generation processing 110a, and the analysis data 121 is output ( B).

Template selection/execution systems 112 and 113 select templates based on the analysis template table from the feature values generated by the feature value generation system 110 (C). That is, an analysis flow corresponding to the feature quantity is selected.

The template selection/execution system 112/113 receives the analysis data 121 output from the feature quantity generation system 110, selects a template 112a from the feature quantity (analysis flow corresponding to the feature quantity), selects the template (selection (C・D).

The output of template selection/execution systems 112 and 113 is sent to an output device by analysis result report output program 114 .

FIG. 5 is a flowchart of feature generation in the data asset analysis support system 104. As shown in FIG. This processing is realized by executing the feature amount generation program 110 by the processing unit composed of the CPU 106 .

First, it is assumed that the input data is restricted such that the first column contains date and time items, which are data relating to date and time, and the second column contains data that includes a failure presence/absence flag, failure location, or other information that serves as an objective variable. The date and time data in the first column is rearranged (sorted) in chronological order (S501). In this embodiment, based on the correlation between date and time and data values, the first feature amount (feature Calculate quantity 1).

FIG. 7 shows an example of input data after rearrangement of the date and time items in the first column among the input data items in step S501.

As shown in FIG. 7, the first column 701 of items is an item related to date and time. The second column is the information on failure occurrence, and the third to fifth columns (703 to 705) are the sensor values assigned to each device. Each device is given a name such as EquipmentA as a name for identifying the device. Further, Sensor1 and the like are attached as sensor identifiers for identifying the sensors attached to each device. The information for identifying the device may be an identifier or the like in addition to the name, and the identifier of the sensor may be the sensor name. The input data may be data related to device failures, as well as business data such as cancellation predictions. In that case, the 2nd column will be the cancellation flag, and the 3rd and subsequent columns will be the contractor information (gender, annual income, etc.).

Furthermore, the sixth and subsequent columns (707 to 709) show the values of each sensor. A sensor name is attached to identify the sensor. FIG. 7 shows an example of the input data in the case of failure predictive diagnosis, but input data for failure location estimation may also be used. In this case, an error code or the like may be entered as an item.

Returning to FIG. 5, the difference between the values in each row of the rearranged date and time items in the first column is calculated (S502). Next, the mode of difference is calculated (S503). Next, it is determined whether the value that does not correspond to the mode of the difference is less than 20% of the total (S504), and if it is less than 20%, the process proceeds to step S506, and the input data is data having periodic characteristics. , and the first feature amount is regular. If it is 20% or more in step S504, the input data is assumed to be data having irregular characteristics, and the first feature quantity is assumed to be irregular.

That is, in this embodiment, based on the correlation between the date and time and the data value, in order to efficiently identify the data type from the data whose data type is unknown, the input data are rearranged so that the first column of the input data is The data related to the date and time items are sorted in chronological order, and the difference between the sorted data in the first column, that is, the difference in the date and time data is calculated.

For example, the interval at which the input data appears is calculated to be 1 minute or 1 day. If the mode of difference is 1 minute, it can be estimated that the data appears periodically every minute. Therefore, in step S504, it is determined whether the data not corresponding to the mode is less than 20%. That is, since 80% or more of the data appear every minute, it is determined that the first feature amount is regular. This is because such periodic data can be assumed to correspond to the type of input data, for example, sensor data of a normal state of wind power generation. On the other hand, if the data in the first column is data appearing regardless of the mode value, it can be estimated to be sensor data when an abnormality of a machine or the like is detected, for example.

The threshold value of 20% shown in step S504 of FIG. 5 is merely an example, and is a value that the analyst can change as appropriate according to input data.

FIG. 6 is a flowchart of feature quantity generation in the data asset analysis support system, and shows the processing continued from FIG.

A step S601 decides whether or not the first feature amount is regular. If the first feature amount is not regular, the process ends.

If the first feature value is regular, the process proceeds to step S602 to extract items whose value types are 100 or less for each data item. The data items are values set by the user, such as the program ID that identifies the program and the operation mode. In the case of a program ID, for example, when the ID value such as "1", "2", and "5" is 100 or less, it is extracted.
In addition to the data item set by the user, if the data item is temperature, there are values such as 10 degrees, 11 degrees, and 15 degrees, and these temperature values may be extracted if they are 100 or less. Also, although there are values of 1 m/s ² and 2 m/s ² from the acceleration sensor, the values of these acceleration sensors may be extracted when they are 100 or less.

Taking the acceleration sensor as an example, if the number of types of values is 100 or less, it can be assumed that the machine repeats a certain operation, which is a kind of mode setting operation. If the value of the acceleration sensor exceeds 100, various numerical values (types) are measured, so it can be assumed that the machine is not in a mode that repeats a fixed operation. Therefore, in this embodiment, in step S602, items with 100 or less types of values for each data item are extracted. be.

In step S603, the mode of duration is calculated for each value. For example, if the extracted item is the output of the acceleration sensor, the duration of the acceleration sensor value of 1m/ ^s2 is 10 times for 1 second, 2 times for 3 seconds, and the duration of 1 second is the most common. 1 second is the mode.

In step S604, it is determined whether or not the value that does not correspond to the mode is less than 20% of the total. For example, taking the above acceleration sensor as an example, if the value other than 1 second is 20% or more, it is determined that there is no second feature amount (feature amount 2) (S605), and if it is less than 20%, the second 2 is determined to be present (S606). That is, as the second feature amount, it is estimated that the target data is data obtained by sensing a machine or the like that repeats a certain motion.

According to the flow of feature amount generation in FIGS. 5 and 6, when the input data has the first feature amount, that is, when the first feature amount is regular, for example, the machine that is the sensing target of the sensor data normally Since there is a high possibility that it is working, it is judged that the predictive diagnosis analysis flow is suitable. If there is no first feature amount, it is highly likely that the machine, which is the sensing target of the sensor data, is not operating normally. Therefore, it is determined that the classification diagnosis analysis flow is suitable.
In the above, sensor data such as temperature and acceleration are used as examples for understanding the invention. It can be adapted by using a value such as a program ID instead of a data value.

For the analysis flow of predictive diagnosis, for example, the analysis described in JP-A-2016-33778 can be applied. Further, as an analysis flow for classification diagnosis, for example, the analysis for classification diagnosis described in Japanese Patent Application Laid-Open No. 2018-25928 can be applied.

If there is a first feature amount, furthermore, from the second feature amount, for example, if the machine to be sensed is operating with a constant repetition of operations, a predictive diagnosis (with mode) analysis flow Alternatively, if the sensing target repeats a certain motion and is not motion, it is determined whether to use the analysis flow of predictive diagnosis (no mode).

5 and 6 are stored in the analysis data 121 by the feature generation program 110. FIG. That is, the feature amount generated by the feature amount generation program 110 is stored in the analysis data table shown in FIG. 9B corresponding to each input data.

FIG. 8 is a flow chart of template selection and execution in the data asset analysis support system. The template selection program 112 reads the analysis data table shown in FIG. 9B (S802). Step S803 starts a loop that repeats the number of records. A step S804 selects from the analysis template table a template that matches the first feature amount (feature amount 1) and the second feature amount (feature amount 2).

The analysis template table, as shown in FIG. 9A, is a table stored in the template 122, storing each feature quantity and analysis flow in association with each other. For example, for data in which feature quantity 1 is regular and feature quantity 2 is absent, an analysis flow for predictive diagnosis (no mode) is stored correspondingly.

Next, the selected template, that is, the analysis flow is executed on the input data (S805). The visualization result of the input data and the template execution result by the visualization execution program 115 are output to a report (S806). When all records have been processed (S807), the process ends.

FIG. 9A is a diagram showing an analysis template table. The analysis template table associates and manages data feature amounts and analysis flows. For example, when the feature quantity 1 is regular and the feature quantity 2 is absent, an analysis flow for predictive diagnosis (no mode) is stored correspondingly.

FIG. 9B is a diagram showing an analysis data table. The analysis data table stores feature amounts calculated for each input data input in a file format in association with input file names. For example, with respect to the input file name [111.csv], the feature quantity 1 is stored as [regular] and the feature quantity 2 is stored as [none].

FIG. 10 is a diagram showing an example of an analysis result report and an output image of the visualization execution program. An analysis result report is displayed by an output device.

FIG. 10(a) is an image of an analysis result report output as a histogram. For example, the x-axis shows the sensor values of the input data shown in FIG. 7, and the y-axis shows the number of data (appearance frequency) corresponding to the x-axis value. Such a histogram is suitable for understanding data distribution trends.

FIG. 10B is an image of the analysis result report output as a line chart. The x-axis represents time and the y-axis shows, for example, the values of the input data items. Line charts are suitable for grasping trends in data over time.

FIG. 10(c) is an image of the analysis result report output as a scatter diagram (correlation graph). The x-axis indicates the value of the item to be compared, and the y-axis indicates the value of the item. In the scatter diagram, the number of items to be compared is output for the item, and it is suitable for understanding the correlation between items.

FIG. 11 is a diagram showing an output image of data analysis, showing the execution result of the analysis flow.

A target file display field 1101 for displaying target file information of an input file that has undergone an analysis flow, and an analysis template display field 1102 for displaying the execution result of the analysis template are displayed. The target file display column 1101 includes the execution date and time of the analysis flow, the target file name, the data volume of the target file, the number of data items included in the target file, the learning period, the objective variable item, the estimated data cycle, and the like.

The analysis template display column 1102 includes a template name 1103, an execution column 1104 indicating which template has been executed, and a result column 1105 indicating the result of executing the template. The example of FIG. 11 shows that the template name [classification diagnosis.knr] was executed and the result was [Accuracy 0.7]. This result shows the certainty of the result of the analysis flow included in the template, and [Accuracy 0.7] shows the accuracy rate = 70% in the test based on the past data of the analysis model.

FIG. 12 is a diagram showing an output image of data analysis. FIG. 12 shows an anomaly degree graph as an output image. FIG. 12 shows that the degree of anomaly increased around July 2016, and signs of failure appeared.

Through the above processing, even if the data owner and the data analyst are different and the data analyst does not sufficiently disclose the information on the data provided by the data owner, pre-analysis of the data to be analyzed, which is the input data , based on the relationship between the date and time and the value of the data, the first feature value that indicates whether the data is regular, the type of value of the data item, and the duration of the data. Since the analysis flow is specified based on the second feature value that indicates whether the target is operating in a certain mode, the number of man-hours required for pre-analysis of data can be reduced, and the time required for data analysis to solve business problems can be significantly shortened. can be shortened to

Also, an appropriate analysis flow can be selected according to data characteristics.

Moreover, according to the present embodiment, results can be obtained easily, so it is possible to motivate users to accumulate data.

104: Data Asset Analysis Support System, 105: Interface, 106: CPU, 107: Storage Device, 108: Memory, 110: Feature Generation Program, 111: Template Management Program, 112: Template Selection Program, 113: Template Execution Program, 114: analysis result report output program, 115: visualization execution program, 120: input data, 121: analysis data, 122: template, 130: input/output device.

Claims (6)

In a data asset analysis support system having a processing unit, a storage device, and an output device,
The storage device stores input data provided by a data owner and an analysis template that stores feature amounts of analysis target data and analysis flows in association with each other,
The processing unit is
Sorting the date and time items of the input data stored in the storage device in chronological order,
Calculate the difference between the date and time items of the input data,
Calculate the mode of the calculated difference,
If the value that does not correspond to the mode value of the calculated difference is less than the threshold value, determine that the first feature amount of the input data is regular,
If the value that does not correspond to the mode value of the calculated difference is not less than a threshold value, determining that the first feature amount of the input data is irregular and generating a feature amount,
selecting an analysis flow for the input data based on the generated feature amount and the analysis template;
executing the selected analysis flow on the input data;
sending the analysis result of the analysis flow to the output device;
The data asset analysis support system, wherein the output device outputs analysis results according to the analysis flow. The processing unit is
when the first feature amount of the input data is regular, extracting items whose value types are equal to or less than a predetermined value for each data item;
calculating the mode of the duration for each value of each data item;
If the value that does not correspond to the mode value of the duration time is less than a predetermined threshold value, it is determined that there is a second feature amount,
2. The data asset analysis support system according to claim 1, wherein when the value not corresponding to the mode of the duration is not less than a predetermined threshold value, it is determined that the second feature quantity does not exist. 3. The data asset analysis support according to claim 2 , wherein the analysis template stored in the storage device stores an analysis flow corresponding to the first feature amount and the second feature amount. system. The storage device stores an analysis data table that associates and stores the first feature amount of the input data, the second feature amount of the input data, and the file name of the input data. 4. The data asset analysis support system according to claim 3 . A data analysis method for a data asset analysis support system having a processing unit, a storage device, and an output device,
The storage device stores input data provided by a data owner and an analysis template that stores feature amounts of analysis target data and analysis flows in association with each other,
The processing unit is
Sorting the date and time items from the input data stored in the storage device in chronological order,
Calculate the difference between the date and time items of the input data,
Calculate the mode of the calculated difference,
If the value that does not correspond to the mode value of the calculated difference is less than the threshold value, determine that the first feature amount of the input data is regular,
If the value that does not correspond to the mode value of the calculated difference is not less than a threshold value, determining that the first feature amount of the input data is irregular and generating a feature amount,
selecting an analysis flow for the input data based on the generated feature amount and the analysis template;
executing the selected analysis flow on the input data;
sending the analysis result of the analysis flow to the output device;
A data analysis method, wherein the output device outputs an analysis result obtained by the analysis flow. The processing unit is
when the first feature amount of the input data is regular, extracting items whose value types are equal to or less than a predetermined value for each data item;
calculating the mode of the duration for each value of each data item;
If the value that does not correspond to the mode value of the duration time is less than a predetermined threshold value, it is determined that there is a second feature amount,
6. The data analysis method according to claim 5, wherein when the value not corresponding to the mode of the duration is not less than a predetermined threshold value, it is determined that the second feature quantity does not exist.

Images