KR20220043314A - Data Pre-Processing System - Google Patents

Abstract

According to one embodiment of the present invention, provided is a data preprocessing system, which includes: a data management portion for storing collected data collected by a data collecting device; and a project management portion for generating a project which is a preprocessing workspace and providing a plurality of preprocessing modules which modularize a preprocessing process of the collected data into a unit function. The project management portion stores unit history data corresponding to the preprocessing modules constituting the preprocessing process of the collected data, and performs the preprocessing process of the collected data according to the preprocessing modules corresponding to the unit history data.

Description

Data Pre-Processing System

The present invention relates to a data pre-processing system, and more particularly, to a data pre-processing system for data analysis.

Data analysis can generally be done in the following order: data collection, data exploration, data preprocessing, modeling, modeling validation, and model deployment. In this case, the data preprocessing process consumes the most time and cost among the data analysis processes.

In the conventional data preprocessing system, a program directly coded based on the analyst's knowledge and experience must be input. Accordingly, the preprocessing result may vary greatly depending on the analyst's knowledge and experience. In addition, even if the analyst has knowledge and experience, a considerable amount of time is consumed to perform pre-processing according to the purpose of data analysis.

The present invention is to solve the above-described problem, and is characterized by providing an abnormal control data detection system capable of generating a normal control data security rule using density estimation.

In addition, the present invention has a technical feature to provide an abnormal control data detection system capable of automatically generating a normal control data security rule for detection of abnormal control data by learning input control data.

A data pre-processing system according to an embodiment of the present invention includes: a data management unit for storing the collected data collected by the data collecting device; and a project management unit that creates a project that is a pre-processing workspace and provides a plurality of pre-processing modules in which the pre-processing of the collected data is modularized into unit functions, wherein the project management unit configures the pre-processing of the collected data It is characterized in that the unit history data corresponding to the pre-processing modules are stored, and the pre-processing of the collected data is performed according to the pre-processing modules corresponding to the unit history data.

According to an embodiment of the present invention, through the pre-processing history presented by the advanced analyst, the pre-processing time can be shortened and even a novice analyst or ordinary person can perform the pre-processing operation.

In addition, according to an embodiment of the present invention, it is possible to reduce the time and cost required to preprocess data by reducing trial and error and repetitive work using the preprocessing history.

In addition, according to an embodiment of the present invention, the preprocessed data may have uniform quality by using the stored preprocessing history for other data preprocessing tasks.

1 is a block diagram schematically showing a data pre-processing system according to an embodiment of the present invention.
2 is a block diagram schematically showing a project history management unit according to an embodiment of the present invention.
3 is a view showing unit history data according to an embodiment of the present invention.
4 is a view showing history data to which the unit history data shown in FIG. 3 is connected.
5A to 5F are diagrams illustrating a history data management method according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

The meaning of the terms described in this specification should be understood as follows.

The singular expression is to be understood as including the plural expression unless the context clearly defines otherwise, and the terms "first", "second", etc. are used to distinguish one element from another, The scope of rights should not be limited by these terms.

It should be understood that terms such as “comprise” or “have” do not preclude the possibility of addition or existence of one or more other features or numbers, steps, operations, components, parts, or combinations thereof.

The term “at least one” should be understood to include all possible combinations from one or more related items. For example, the meaning of “at least one of the first, second, and third items” means that each of the first, second, or third items as well as two of the second, second, and third items It means a combination of all items that can be presented from more than one.

1 is a diagram showing a data pre-processing system according to the present invention.

As shown in FIG. 1 , the data preprocessing system 1000 according to the present invention includes a data management unit 100 and a project management unit 200 .

The data management unit 100 stores and manages pre-processing target data. Here, the pre-processing target data may be data collected by a data collection device (not shown). In an embodiment, the data collection device may collect micro data generated in the course of various processes. Here, the micro data refers to raw data as data itself collected through various sensors and the like. Hereinafter, for convenience of description, micro data will be referred to as collected data.

The data collection device includes various instruments, sensors, actuators, and the like for collecting micro data. The data collection device may further include a P/C, a programmable logic controller (PLC), a distributed control system (DCS), etc. that integrate or control data collected by a measuring instrument, a sensor, an actuator, and the like.

For example, the data collection device may collect data generated in a continuous process. The continuous process refers to a process in which a plurality of processes for producing a finished product using raw materials are continuously performed, and the outputs of each process are mixed with each other or the state of the output of a specific process is changed and supplied to a subsequent process. Steel processing is a representative example of such a continuous process. The steel process may include various processes such as a steelmaking process, a steelmaking process, a casting process, and a rolling process. When the data collection device is applied to a steel process, it is possible to collect micro data generated in the course of various processes such as a steelmaking process, a steelmaking process, a casting process, and a rolling process.

The data management unit 100 includes a data storage unit 110 and a data information editing unit 120 .

The data storage unit 110 stores pre-processing target data, for example, collected data collected by the data collection device.

The data information editing unit 120 edits information such as the format and authority of the collected data stored in the data storage unit 110 . The data information editing unit 120 may edit information such as the format and authority of the collected data according to a user input.

The project management unit 200 creates a project that is a pre-processing work space, and manages at least one generated project. The project management unit 200 includes a project creation unit 210 , a project information editing unit 220 , and a project history management unit 230 .

The project generating unit 210 creates a project that is a pre-processing work space. The project generator 210 may create a new project according to a user's request. The project generator 210 may provide various functions so that the user can easily perform data pre-processing in the created project.

In one embodiment, the project generating unit 210 may perform data pre-processing on the collected data according to the edited history data in the project history management unit 230 to be described later. For example, history data corresponding to the preprocessing module sequentially operated in the first project by the project history management unit 230 is copied to the second project, and the project generating unit 210 is copied to the second project in the first project. According to the historical data of the project, the data can be pre-processed and the result can be calculated. Alternatively, the project generating unit 210 may perform data pre-processing according to the history data added, modified, deleted, inactive, or reordered by the project history management unit 230 to recalculate the result. Accordingly, it is possible to reduce the time and cost required for data preprocessing by reducing trial and error and repetitive work by using the data preprocessing history of an experienced analyst. In addition, data of uniform quality can be generated.

The project information editing unit 220 edits information such as the name and authority of the project created by the project generation unit 210 . The project information editing unit 220 may edit information such as a name and authority of a project according to a user input.

The project history management unit 230 manages the history data of sequentially operating the pre-processing module in the project.

The project history management unit 230 will be described in detail with reference to FIGS. 2 to 5F . 2 is a block diagram schematically showing a project history management unit according to an embodiment of the present invention. 3 is a view showing unit history data according to an embodiment of the present invention, and FIG. 4 is a view showing history data to which the unit history data shown in FIG. 3 is connected. 5A to 5F are diagrams illustrating a history data management method according to an embodiment of the present invention.

As shown in FIG. 2 , the project history management unit 230 includes a history generation unit 231 , a history addition unit 232 , a history correction unit 233 , a history deletion unit 234 , a history inactivation unit 235 , It includes a history copy unit 236 and a history order change unit 237 .

The history generating unit 231 generates history data in which the preprocessing modules are sequentially operated. Specifically, whenever the pre-processing module is operated, the history generating unit 231 generates unit history data corresponding to the corresponding pre-processing module as shown in FIG. 3, and as the pre-processing module operates sequentially, as shown in FIG. 4 And as shown in FIG. 5A , the history data 300 is generated by sequentially connecting the generated unit history data 301 . In this case, the history data 300 includes at least one unit history data 301 .

Referring to FIG. 3 , the unit history data 301 includes a head part H, a data part D, and a tail part T. As shown in FIG.

Information of the unit history data 301 preceding the corresponding unit history data 301 is stored in the head unit H. For example, an address in which unit history data 301 preceding the corresponding unit history data 301 is stored may be stored in the head unit. Alternatively, when the corresponding unit history data 301 is the first unit history data, 'Null' is stored in the head unit.

Information of the corresponding unit history data 301 is stored in the data part (D). Specifically, the data unit D includes a first data unit for storing the status code (Status) of the unit history data 301, and request data (Request) for performing data pre-processing using the unit history data 301 . ) and a second data part for storing the response data (Response) and a third data part for storing the preprocessing module number (Function_id) corresponding to the unit history data 301 . In this case, the status code of the corresponding unit history data 301 stored in the first data unit may be a code indicating the status of each unit history data 301 as shown in Table 1. In addition, the preprocessing module number (Function_id) stored in the third data unit may be a preprocessing module number corresponding to each preprocessing module operated in the project as shown in Table 2. In particular, the pre-processing module list is stored in the project information editing unit 220, and the pre-processing module number (Function_id) of the third data part is any one of the pre-processing module lists stored in the project information editing unit 220 pre-processing module of the list of pre-processing modules. It can match any number.

status code state 0 Ready One Running 2 Done 3 Error 4 Delete 5 Disable 6 Initialized

function_id Functions of the preprocessing module One data load 2 Change variable name and type 3 Automatic quality diagnosis report output 4 Selective quality diagnosis 6 percentage of characters in a variable 7 percentage of spaces within a variable 8 Check and remove data duplication 9 Correlation Coefficient Reporting 10 data visualization 11 Automate data cleansing 13 Remove missing values 14 Remove outliers 15 delete variable 16 Normalized Transform 17 Creating a cumulative sum variable 18 Split Variables 19 Combining Variables 20 Variable categorization 21 create dummy variable 22 Create an interaction variable 23 Create maximum correlation lag variable 24 Outlier replacement 25 Replace missing values 26 data merge 27 Influencer detection 28 Bulk Category Clustering 29 Staging Variable Suggestion

Information of the unit history data 301 following the corresponding unit history data 301 is stored in the tail part T. For example, an address in which unit history data 301 following the corresponding unit history data 301 is stored may be stored in the tail portion T. When the corresponding unit history data 301 is the last unit history data 301, 'Null' is stored in the tail part T.

In one embodiment, the plurality of unit history data 301 are connected in a linked list structure sequentially connected through the head part H and the tail part T of each unit history data to provide the history data. make up

As shown in FIG. 5B , the history adding unit 232 may add unit history data 304 to the generated history data 300 . Specifically, the history adding unit 232 may add new unit history data 304 to the generated history data 300 . The history adding unit 232 may connect new unit history data 304 that the user wants to add to the last unit history data 303 of the generated history data 300 .

As shown in FIG. 5C , the history correction unit 233 may edit the function of the preprocessing module by modifying the data unit D of the unit history data 301 , 302 , 303 , and 304 . Specifically, the history correction unit 233 may modify the data unit D of the unit history data constituting the history data 300 to correct detailed functions of the preprocessing module corresponding to the unit history data 303 .

In one embodiment, the project management unit 200 may perform data pre-processing on the collected data by operating a pre-processing module corresponding to each unit history data based on the history data including the unit history data in which the data unit is modified. .

The history deletion unit 234 may delete unit history data constituting the history data. Specifically, the history deletion unit 234 corrects the first data part of the unit history data to be deleted to a status code (see Table 1) corresponding to Delete, and as shown in FIG. 5D , the history deletion unit At 234, the connection between the unit history data 302 and the unit history data 301 preceding the unit history data 302 and the unit history data 303 following the unit history data 302 is released, and the unit history data 302 ), the unit history data 301 preceding the unit history data 302 is connected with the unit history data 303 following the corresponding unit history data 302 .

In one embodiment, the project management unit 200 may perform data pre-processing of the collected data by operating a pre-processing module corresponding to each unit of history data based on the history data from which at least one unit of historical data is deleted.

The history deactivation unit 235 may inactivate unit history data constituting the history data. For example, when it is necessary to compare the result of preprocessing according to the user's current history with the result of preprocessing according to the history of deleting the unit history data, the result may be compared by performing deactivation on the corresponding unit history data. Specifically, the history deactivation unit 235 modifies the first data part of the unit history data to be deactivated to a status code corresponding to Disable, and deactivates the unit history data as shown in FIG. 5E . When data pre-processing is performed according to the corresponding history data, the function of the pre-processing module corresponding to the inactive unit history data is not performed, but the function of the pre-processing module corresponding to the following unit history data is performed.

After pre-processing according to the history data, the inactive unit history data may be restored by modifying the first data part to a status code corresponding to Done.

The history copy unit 236 may copy history data for a pre-processing process performed in one project to another project. Specifically, the history copy unit 236 may generate second history data by copying the first history data generated as the pre-processing modules sequentially operate in the first project to the second project. Accordingly, it is possible to reduce the time and cost required to pre-process data by reducing trial and error and repetitive work using the pre-processing history data that has already been performed.

In one embodiment, the project management unit 200 may perform data preprocessing on the collected data by operating a preprocessing module corresponding to each unit of history data based on the copied history data.

The history order changing unit 237 may change the order of unit history data constituting the history data. For example, as shown in FIG. 5F , the history order changing unit 237 may change the order of the second unit history data and the third unit history data. After disconnecting the second unit history data and the third unit history data, connecting the second unit history data to the fourth unit history data, and connecting the third unit history data to the first unit history data, the third unit history data and the second unit history data By linking data, the order of the second unit history data and the third unit history data can be changed.

In one embodiment, the project management unit 200 may perform data pre-processing on the collected data by operating a pre-processing module corresponding to each unit history data based on the history data in which the order of the unit history data is changed.

According to an embodiment of the present invention, the project management unit 200 pre-processes the collected data according to the pre-processing module corresponding to each unit history data for the history data including the unit history data edited by the project history management unit 230 . can do. Accordingly, the pre-processing time can be shortened by using the pre-processing history data presented by the advanced analyst, and even a novice analyst or the general public can perform the pre-processing work.

In addition, according to an embodiment of the present invention, data preprocessed by using the stored history data in another project may have uniform quality.

Those skilled in the art to which the present invention pertains will understand that the above-described present invention may be embodied in other specific forms without changing the technical spirit or essential characteristics thereof.

Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

All disclosed methods and procedures described herein may be implemented, at least in part, using one or more computer programs or components. These components may be configured as a series of series via any conventional computer readable medium or machine readable medium including volatile and nonvolatile memory such as RAM, ROM, flash memory, magnetic or optical disks, optical memory, or other storage media. They may be provided as computer instructions. The instructions may be provided as software or firmware, and may be implemented, in whole or in part, in a hardware configuration such as ASICs, FPGAs, DSPs, or any other similar device. The instructions may be configured to be executed by one or more processors or other hardware components, which when executing the series of computer instructions perform all or part of the methods and procedures disclosed herein or make it possible to do

1000: data pre-processing system 100: data management unit
110: data storage unit 120: data information editing unit
200: project management unit 210: project storage unit
220: project editing unit 230: project history management unit

Claims (9)

a data management unit for storing the collected data collected by the data collecting device; and
A project management unit that creates a project that is a pre-processing workspace and provides a plurality of pre-processing modules that modularize the pre-processing process of the collected data into a unit function; includes;
The project management unit,
Storing unit history data corresponding to the pre-processing modules constituting the pre-processing of the collected data,
Data pre-processing system, characterized in that the pre-processing of the collected data is performed according to pre-processing modules corresponding to the unit history data. According to claim 1,
The data pre-processing system, characterized in that the unit history data corresponding to the pre-processing modules constituting the pre-processing of the collected data constitutes the history data of the pre-processing of the collected data. According to claim 1,
The unit history data data pre-processing system, characterized in that for storing the state information indicating the state of the pre-processing module corresponding to the unit history data and the module number indicating the function of the pre-processing module. According to claim 1,
The project management unit,
Data pre-processing system, characterized in that by editing the unit history data constituting the history data to correct the function of the pre-processing module corresponding to the unit history data, and pre-processing the collected data according to the function-modified pre-processing module. According to claim 1,
The project management unit,
As the pre-processing module sequentially operates with respect to the collected data in the project, a project history management unit that generates unit history data corresponding to the sequentially operated pre-processing module, and generates history data using the generated unit history data ; Data pre-processing system comprising a. According to claim 1,
The project management unit,
The first history data generated as the preprocessing modules sequentially operate in the first project are copied to the second project to generate second history data, and in the second project, each of the unit history data constituting the second history data is Pre-processing the collected data according to corresponding pre-processing modules. According to claim 1,
The project management unit,
The data preprocessing system, characterized in that adding, deleting, and changing the order of the unit history data using a connection between the unit history data. According to claim 1,
The history data is composed of unit history data corresponding to the pre-processing module,
The unit history data is
a head unit for storing information in which unit history data preceding the corresponding unit history data is stored;
a data unit for storing information on a pre-processing module corresponding to the unit history data; and
Data pre-processing system comprising a; a tail unit for storing information on unit history data that follows the unit history data. 9. The method of claim 8,
The history data is connected in a linked list structure using a head portion and a tail portion of the unit history data corresponding to the pre-processing module to constitute the history data.

Images