KR102706053B1 - A variable selection device based on feature selection algorithm and a model management system - Google Patents

Abstract

A variable selection device according to one embodiment of the present invention may include a variable selection device that determines a representative variable, which is a variable representing the characteristics of a cloud server, wherein the variable selection device comprises: a selection reception module that receives operation information, which is information about each of a plurality of variables, as information generated when the cloud server operates; a detection module that analyzes, based on the operation information, a causality between a target variable, which is a predetermined variable, and a comparison variable, which is a variable excluding the target variable, according to a predetermined judgment method; and a determination module that determines the comparison variable and the target variable, which have a causality satisfying a predetermined causality condition, as the representative variable.

Description

{A VARIABLE SELECTION DEVICE BASED ON FEATURE SELECTION ALGORITHM AND A MODEL MANAGEMENT SYSTEM}

The present invention relates to a variable selection device that selects variables used in a machine learning model by utilizing a feature selection algorithm, and to a model management system including the same.

With the introduction of the cloud, many services have benefited from efficient network load response, rapid recovery in case of failure, and flexible server capacity calculation. At the same time, infrastructure managers are required to understand and respond to not only simple servers but also the complex environment of cloud systems. There are many limitations to responding to these issues through individual manager judgment or rule-based processes, and thus the concept of AIOps (Artificial Intelligence for IT Operations) has emerged. This is the introduction of artificial intelligence to IT operations, and it is mainly moving toward efficiently optimizing data centers, increasing customer satisfaction, and improving development productivity. From a research perspective, algorithms that calculate the appropriate capacity of resources and proactively respond to failures are considered key elements.

In order to maximize the performance of the above algorithms, it is important to analyze all kinds of data (Metric, Log, Trace) collected from the data center and predict what values will be shown in the future. By predicting resource usage (Metric), it is possible to calculate the appropriate capacity of resources by considering user usage patterns for several months in the future. In addition, by predicting resource usage and patterns of logs that are likely to occur in the future, it is possible to proactively detect patterns related to failures and analyze the root cause to shorten the resolution time. However, as the environment becomes more complex due to the introduction of the cloud, data pipelines have also become more sophisticated and complex, requiring a lot of data, time, and resources to learn machine learning models for failure detection or future workload prediction. As a result, although accuracy has increased, server overload problems have occurred due to learning machine learning models.

The present invention is intended to solve the above-described problem, and provides a model management system including a variable selection device for selecting only key variables required for a model and a timing determination device for determining a learning timing of the model.

A variable selection device according to one embodiment of the present invention may include a variable selection device that determines a representative variable, which is a variable representing the characteristics of a cloud server, wherein the variable selection device comprises: a selection reception module that receives operation information, which is information about each of a plurality of variables, as information generated when the cloud server operates; a detection module that analyzes, based on the operation information, a causality between a target variable, which is a predetermined variable, and a comparison variable, which is a variable excluding the target variable, according to a predetermined judgment method; and a determination module that determines the comparison variable and the target variable, which have a causality satisfying a predetermined causality condition, as the representative variable.

In addition, the above-determined judgment method may be a method of judging causality between two variables using the Granger causality test.

In addition, the method further includes a search module that detects whether an event has occurred in the operation information of the target variable by distinguishing and analyzing the operation information of the target variable for each predetermined time range, and the detection module can analyze causality by comparing the operation information of the target variable in which an event has occurred and the operation information of the comparison variable in the same time range according to the predetermined judgment method.

In addition, the above-determined causality condition may be a condition in which the above-determined time range in which causality is recognized is the largest.

In addition, the above-described selection receiving module further includes a preprocessing module that preprocesses the motion information received according to a predetermined preprocessing method; the detection module analyzes the causality between the target variable and the comparison variable based on the preprocessed motion information, and the predetermined preprocessing method may be a method of removing the motion information that is categorical or has a variance below a predetermined standard.

In addition, the above-determined preprocessing method may be a method of removing the operation information for a variable whose correlation coefficient with the target variable is higher than a predetermined standard.

A variable selection method according to one embodiment of the present invention is implemented by a variable selection device, and may include a step of receiving, by a selection receiving module, operation information, which is information about each of a plurality of variables - the variables including a target variable, which is a predetermined variable, and a comparison variable, which is a variable excluding the target variable - as information generated while the cloud server is operating; a step of distinguishing and analyzing, by a search module, the operation information of the target variable for each predetermined time range, and detecting whether an event has occurred in the operation information of the target variable; a step of analyzing, by a detection module, the operation information of the target variable in which an event has occurred and the operation information of the comparison variable that has occurred in the same time range according to a predetermined judgment method, and analyzing causality; and a step of determining, by a decision module, the comparison variable and the target variable having causality satisfying a predetermined causality condition as the representative variable.

According to one embodiment of the present invention, a model management system for managing a machine learning model operated in a cloud server includes: a variable selection device for determining a representative variable, which is a variable representing a characteristic of the cloud server, based on operation information, which is information about each of a plurality of variables generated as information when the cloud server is operated; and a time determination device for determining a time when the machine learning model is re-learned based on the operation information about the representative variable; wherein the variable selection device may include a selection reception module for receiving the operation information, a detection module for analyzing causality between a target variable, which is a predetermined variable, and a comparison variable, which is a variable excluding the target variable, according to a predetermined judgment method based on the operation information, and a determination module for determining the comparison variable and the target variable, which have causality satisfying a predetermined causality condition, as the representative variable.

In addition, the timing determination device includes a timing receiving module that receives the motion information in time series, a seasonal module that decomposes the motion information in time series and then calculates the seasonal intensity of the motion information according to a predetermined intensity analysis method, a pattern module that analyzes a pattern of the motion information using a predetermined pattern analysis method, and a judgment module that determines the re-learning timing of the machine learning model performed in the cloud server according to a predetermined decision method according to the magnitude of the seasonal intensity, and the predetermined decision method may be a method of determining the re-learning timing by considering the seasonality of the motion information or the pattern of the motion information.

In addition, the above-determined determination method may include a first determination method for determining a re-learning time by considering the seasonality of the motion information when the seasonality intensity of the motion information is equal to or greater than a first reference value, and a second determination method for determining a re-learning time by considering the pattern of the motion information when the seasonality intensity of the motion information is less than a second reference value that is lower than the first reference value.

A variable selection device according to the present invention and a model management system including the same can maximize the utilization efficiency of server resources.

Additionally, it can prevent server downtime.

Additionally, it can improve the performance of the model.

However, the effects of the present invention are not limited to the effects described above, and effects not mentioned can be clearly understood by a person having ordinary skill in the art to which the present invention pertains from this specification and the attached drawings.

Figure 1 is a schematic diagram of a model management system according to one embodiment of the present invention.
Figure 2 is a configuration diagram of a model management system according to one embodiment of the present invention.
Figure 3 is a configuration diagram of a variable selection device of a model management system according to one embodiment of the present invention.
Figure 4 is a flow chart of a variable selection method performed by a variable selection device according to one embodiment of the present invention.
Figure 5 is an operation diagram of a variable selection device according to one embodiment of the present invention.
Figure 6 is a representative variable selected by a variable selection device according to one embodiment of the present invention.
Figure 7 is a configuration diagram of a timing determination device of a model management system according to one embodiment of the present invention.
Figure 8 is a flow chart of a timing determination method of a timing determination device according to one embodiment of the present invention.
Figure 9 is an operation diagram of a timing determination device according to one embodiment of the present invention.
FIG. 10 is a graph for explaining the first decision method executed by the judgment module of the timing decision device according to one embodiment of the present invention.
FIG. 11 and FIG. 12 are drawings for explaining the third decision method executed by the judgment module of the timing decision device according to one embodiment of the present invention.

Hereinafter, specific embodiments of the present invention will be described in detail with reference to the drawings. However, the spirit of the present invention is not limited to the presented embodiments, and those skilled in the art who understand the spirit of the present invention will be able to easily propose other regressive inventions or other embodiments included within the scope of the spirit of the present invention by adding, changing, deleting, etc. other components within the scope of the same spirit, but this will also be considered to be included within the scope of the spirit of the present invention.

FIG. 1 is a schematic diagram of a model management system according to one embodiment of the present invention.

Referring to FIG. 1, a model management system (100) according to one embodiment of the present invention can be connected to a cloud server (S10) via a wired/wireless network to enable information communication.

The wireless network referred to in the present invention may be a core network integrated with a wired public network, a wireless mobile communication network, or a portable Internet, or may mean a worldwide open computer network structure that provides various services existing in the TCP/IP protocol and its upper layer, such as HTTP (Hyper Text Transfer Protocol), HTTPS (Hyper Text Transfer Protocol Secure), Telnet, FTP (File Transfer Protocol), DNS (Domain Name System), SMTP (Simple Mail Transfer Protocol), and the like, and is not limited to these examples, but comprehensively means a data communication network capable of transmitting and receiving data in various forms.

A cloud server (S10) may mean a server that provides cloud services to a client (C10).

The server referred to in the present invention may include other configurations for performing a server environment of the server. The server may include any type of device.

For example, a server may be a digital device having computing power, such as a laptop computer, a notebook computer, a desktop computer, a web pad, or a mobile phone, and having a processor and memory.

For example, the server may be a web server. However, the type of server is not limited to this, and can be changed in various ways at a level that is obvious to a person skilled in the art.

For example, a cloud server (S10) can provide cloud services to clients by configuring a virtual machine or container for each client (C10) and providing necessary operations or storage according to the client's request.

The model management system can be connected to at least one cloud server.

The model management system (100) can manage and operate a machine learning model running on a cloud server.

Below, we will describe the model management system in detail.

Figure 2 is a configuration diagram of a model management system according to one embodiment of the present invention.

Referring to FIG. 2, a model management system (100) according to one embodiment of the present invention may include a variable selection device (110) that determines variables for learning a learning model, which is a model learned by machine learning, and a timing determination device (120) that determines a timing for re-learning the learning model.

In addition, the model management system (100) may further include an interface device (130) that generates a user interface that displays information such as variables or re-learning timing generated by the variable selection device and the timing determination device and transmits the information to the manager.

Here, the administrator may mean a person or a company that manages and operates a cloud server. Alternatively, the administrator may mean a person or a company that manages and operates a model management system.

The timing decision device can perform a process (timing decision method) for determining the timing for re-learning after a certain period of time has passed since the learning model has learned.

For example, the given period of time may be one month, but the present invention is not limited thereto.

As another example, the timing decision device can perform a process (timing decision method) to determine the timing of re-learning when learning data has accumulated beyond a predetermined capacity after the learning model has learned.

For example, the specified capacity may be 100 TB, but the present invention is not limited thereto.

The interface device (130) can produce a user interface that displays the representative variable determined by the variable selection device (110) and the grounds used to determine the representative variable, and transmit the result to the manager.

For example, the user interface can display the values of the target variable's behavior information in a time series graph format and can display the time range in which an event occurred.

For example, an administrator can specify a target variable by entering it into the user interface.

The interface device (130) can generate a user interface that displays the re-learning time determined by the timing determination device (120) and the grounds used to determine the re-learning time, and transmit the result to the administrator.

For example, the user interface could display information about the seasonality of the motion information, what decision method was used, etc.

The term administrator may include the administrator's computing device.

FIG. 3 is a configuration diagram of a variable selection device of a model management system according to one embodiment of the present invention.

Referring to FIG. 3, a model management system according to an embodiment of the present invention may include a variable selection device (110) that determines a representative variable, which is a variable representing the characteristics of a cloud server, a selection reception module (111) that receives operation information, which is information about each of a plurality of variables, as information generated when the cloud server is operated, based on the operation information, a detection module (114) that analyzes causality between a target variable, which is a predetermined variable, and a comparison variable, which is a variable excluding the target variable, according to a predetermined judgment method, and a decision module (115) that determines the comparison variable and the target variable having causality satisfying a predetermined causality condition as the representative variable.

In addition, the variable selection device (110) may further include a search module (113) that distinguishes and analyzes the operation information of the target variable for each predetermined time range and detects whether an event occurs in the operation information of the target variable.

In addition, the variable selection device (110) may further include a preprocessing module (112) that preprocesses the operation information received by the selection receiving module (111) according to a predefined preprocessing method.

The selection receiving module (111) can receive operation information, which is information generated when the cloud server operates.

For example, the motion information may be information about each of multiple variables.

For example, variables can be CPU usage, memory usage, disk read data, disk write data, network traffic, graphics processing unit usage, graphics processing unit memory usage, etc.

For example, there can be a total of 89 variables.

However, without limitation, the total number of variables can be varied in various ways at a level that is obvious to ordinary technicians.

For example, motion information may be time-series data, meaning that values are assigned over time.

The preprocessing module (112) can preprocess the operation information received by the selection receiving module (111) according to a pre-determined preprocessing method.

For example, the above-determined preprocessing method may include a method of removing motion information whose time is less than a predetermined time.

For example, motion information is time-series data, and there may be a time period during which the data exists. If the given time period is 150 days, motion information collected and stored for 30 days may be removed by a pre-determined preprocessing method.

For example, the above-described preprocessing method may further include a method of removing operation information in which the usage rate of a specific variable is below a predetermined rate or the number of virtual machines of a cloud server or processes running on a cloud server is below a predetermined standard.

Specifically, the specific variable may be CPU usage, where the predetermined ratio may be 5%, but the present invention is not limited thereto.

For example, the predetermined standard for the number of processes may be 5, but the present invention is not limited thereto.

When CPU usage is low or the number of running processes (programs) is small, it may be difficult to infer causal relationships based on operational information.

For example, the above-determined preprocessing method may further include a method of removing the operation information for a variable whose correlation coefficient with the target variable is greater than a predetermined standard.

For example, a given criterion for the correlation coefficient might be 0.9.

The correlation coefficient is a number between -1 and 1, and the closer it is to 1, the stronger the correlation.

The meaning of a variable having a high correlation coefficient with the target variable is that it has almost the same meaning as the target variable, so there may be no need to analyze a separate causal relationship.

For example, the above-determined preprocessing method may be a method of removing the motion information having a categorical or variance below a predetermined standard.

Categorical information can mean information that can be divided into categories or groups.

For example, the predetermined criterion for dispersion may be 0.5. However, the present invention is not limited thereto.

The preprocessing method may further include a method of correcting time errors between indicator data indicating operation information of each variable.

For example, the lag value can be calculated as the average of the univariate autocorrelation (ACF) values of the target variable.

However, without limitation, the lag value can be determined in other ways.

For example, the lag value can be determined as the value obtained by multiplying a predefined time range by '3'.

For example, the predefined time range may be 5 minutes. However, this is not limited to this, and the predefined time range may be varied in various ways as would be obvious to a person skilled in the art.

The reason for multiplying by '3' may be because a three-step process is performed by the search module (113), the detection module (114), and the decision module (115).

The search module (113) can detect whether an event occurs in the operation information of the target variable by distinguishing and analyzing the operation information of the target variable at each predetermined time range.

The exploration module (113) can distinguish the operation information (indicator data) of the target variable for each pre-determined time range.

For example, target variables can be CPU usage, memory usage, disk read bytes, disk write bytes, network in bytes, and network out bytes.

However, without limitation to this, the specific type of the objective variable can be varied in various ways at a level that is obvious to ordinary technicians.

The behavioral information of a single variable can be defined as indicator data.

The exploration module (113) can determine whether an event has occurred within the indicator data by utilizing basic statistics such as the distribution of the values of the indicator data.

For example, if the average value of the indicator data over a predetermined time range is greater than a predetermined percentage of the overall average value of the indicator data, the search module (113) can determine that an event has occurred over the predetermined time range.

For example, if the variance of indicator data over a predetermined time range is greater than a predetermined variance value, the search module (113) can determine that an event has occurred in the predetermined time range.

The search module (113) can only perform information processing on preprocessed motion information.

The detection module (114) can analyze the causality between the target variable and the comparison variable based on the preprocessed motion information.

The detection module (114) can analyze causality by comparing the operation information of the target variable where the event occurred and the operation information of the comparison variable that occurred in the same time range as the operation information where the event occurred, according to the predetermined judgment method.

A pre-determined judgment method could be a method of judging causality between two variables using the Granger causality test.

The Granger causality test is a statistical algorithm for assessing the causal relationship between two variables in time series data.

---------------------- [Mathematical Formula 1]

------------[Mathematical Formula 2]

In the above two equations, if it is shown through the p-value that Equation 2 is a better regression model than Equation 1 by performing an F test, etc., it can be defined that x has a Granger causal relationship with y.

Below, a detailed description of the Granger causality test may be omitted within the scope of known technology.

The detection module (114) can check the causality between the indicator data constituting the target variable and the indicator data constituting the comparison variable based on Granger causality and calculate the p-value.

If the p-value (significance probability) of the Granger causality of the indicator data within a predetermined time range based on an arbitrary comparison variable is less than the significance level, the indicator data within the predetermined time range of the comparison variable can be judged to have a causal relationship with the target variable.

A comparison variable can mean any variable except the target variable.

One comparison variable can be judged to have Granger causality with all target variables.

The detection module (114) can generate a p-value matrix using the p-value.

The decision module (115) can determine the comparison variable and the target variable having a causality satisfying a pre-determined causality condition as the representative variable.

The pre-determined causality condition may be the condition with the largest number of pre-determined time periods over which causality is recognized.

The decision module (115) can extract variables that show high causality with a virtual progress by summing them by section using the p-value matrix.

Specifically, the decision module (115) can calculate the number of outputs, which is the number of predetermined time ranges that show causality with the target variable among the predetermined time ranges of one comparison variable.

The decision module (115) can satisfy a pre-determined causality condition by selecting a comparison variable with the highest number of outputs among the comparison variables.

However, the present invention is not limited thereto, and a comparison variable that is calculated more than a predetermined ratio of the number of times an event occurs can satisfy a pre-determined causality condition.

Here, the predetermined ratio may be 80%. However, the present invention is not limited thereto.

The decision module (115) can transmit a list of representative variables to a cloud server.

A cloud server or a virtual machine of a cloud server can learn a future prediction model that predicts a workload expected to occur in the future period using only the operation information of representative variables received from the decision module (115). (Learning model)

For example, information about the specifications of a cloud server, the type of application running on the cloud server, the type of program installed on the cloud server, and the command set on the cloud server are set as input data, and the operation information that occurred in the past labeled in the input data is used as output data, and a future prediction model can be learned through machine learning/deep learning.

At this time, only the motion information for representative variables can be utilized.

In addition, a cloud server or a virtual machine of a cloud server can learn an anomaly detection model that detects anomalies in the cloud server or virtual machine using only representative variables. (Learning model)

For example, information about the specifications of a cloud server, the type of application running on the cloud server, log data and trace data generated from the cloud server, the type of program installed on the cloud server, and commands set on the cloud server are set as input data, and an anomaly detection model can be trained through machine learning/deep learning through data labeled with whether the cloud server is normal/abnormal.

This allows cloud servers or virtual machines to minimize cloud resources used for machine learning and further improve the performance of machine learning models.

Below, the variable selection method implemented by the variable selection device (110) will be described in detail.

Figure 4 is a flowchart of a variable selection method performed by a variable selection device (110) according to one embodiment of the present invention.

Referring to FIG. 4, a variable selection method according to an embodiment of the present invention is implemented by a variable selection device (110), and determines a representative variable, which is a variable representing the characteristics of a cloud server, in a variable selection method, a selection receiving module (111) receives a plurality of variables as information generated while the cloud server is operating - the variables include a target variable, which is a predetermined variable, and a comparison variable, which is a variable excluding the target variable. - It may include a step of receiving the motion information, which is each information about the fields, a step of distinguishing and analyzing the motion information of the target variable for each predetermined time range by the search module (113) and detecting whether an event has occurred from the motion information of the target variable, a step of comparing the motion information of the target variable in which an event has occurred and the motion information of the comparison variable that occurred in the same time range according to the predetermined judgment method by the detection module (114) and analyzing causality, and a step of determining the comparison variable and the target variable having causality that satisfies the predetermined causality condition as the representative variable by the decision module (115).

Below, detailed explanations may be omitted to the extent that they overlap with the above-mentioned contents.

FIG. 5 is an operational diagram of a variable selection device according to one embodiment of the present invention.

Referring to FIG. 5, the selection reception module (111) can receive operation information from the cloud server.

The preprocessing module (112) can remove indicator data for some variables from the operation information received from the above-mentioned selection receiving module (111) according to a preprocessing method determined in advance. (Preprocessing process)

Preprocessed motion information can be transmitted to the search module (113).

The search module (113) can classify the operation information of the target variable into predetermined time ranges and determine whether an event has occurred in units of predetermined time ranges.

For example, referring to Fig. 5(a), the operation information of any one target variable can be divided into the 1-1 section (K11), the 1-2 section (K12), the 1-3 section (K13), the 1-4 section (K14), and the 1-5 section (K15).

At this time, it can be determined that an event occurred in the 1st-2nd section (K12) and the 1st-4th section (K14).

The detection module (114) can calculate a p-value using the Granger causality test for the operation information of the target variable where the event occurred and the operation information of the comparison variable that occurred in the same time range as the operation information where the event occurred.

For example, referring to Fig. 5(b), the operation information of the first comparison variable can be divided into the 2-1 section (K21), the 2-2 section (K22), the 2-3 section (K23), the 2-4 section (K24), and the 2-5 section (K25).

Here, the detection module (114) can determine the 2-2 section (K22) and the 2-4 section (K24), which are sections of the same time zone as the 1-2 section (K12) and the 1-4 section (K14), which are sections of the target variable where the event occurred, as targets of causality analysis.

The motion information of the 2-2 section (K22) can be acknowledged to be causally related to the motion information of the 1-2 section (K12), and the motion information of the 2-4 section (K24) can be acknowledged to be causally related to the motion information of the 1-4 section (K14).

Referring to Fig. 5(c), the operation information of the second comparison variable can be divided into the 3-1 section (K31), the 3-2 section (K32), the 3-3 section (K33), the 3-4 section (K34), and the 2-5 section (K35).

The detection module (114) can determine the 3-2 section (K32) and the 3-4 section (K34), which are sections of the same time zone as the 1-2 section (K12) and the 1-4 section (K14), which are sections of the target variable where the event occurred, as targets of causality analysis.

The motion information of the 3-2 section (K32) may not be recognized as causally related to the motion information of the 1-3 section (K13), and the motion information of the 3-4 section (K34) may not be recognized as causally related to the motion information of the 1-4 section (K34).

The decision module (115) can determine the comparison variable and the target variable having a causality satisfying a pre-determined causality condition as the representative variable.

The decision module (115) can transmit representative variables to a cloud server so that the cloud server can use them for machine learning.

FIG. 6 is a drawing for explaining the effect of a representative variable selected by a variable selection device according to one embodiment of the present invention.

To evaluate the effectiveness of the present invention, RMSE and MAE values were used.

------------ [Mathematical Formula 3]

------------------[Mathematical Formula 4]

For performance evaluation, the predefined time range can be '1 day'.

Referring to Fig. 6(a), when looking at most indicators and prediction models, the present invention shows excellent performance in most cases.

In each prediction model, both RMSE and MAE values show the same tendency in the performance of the prediction model learned as the representative variable. In addition, the VAR prediction model showed the lowest RMSE and MAE values compared to other prediction models. In other words, it was found that the algorithm of the present invention has the greatest effect on the VAR prediction model. It was investigated that the VAR prediction model showed a 67% performance improvement compared to the model with the worst RMSE value. Furthermore, it was investigated that the KNN prediction model showed an 18% performance improvement and the LSTM showed a 17% performance improvement.

Referring to Fig. 6(b), one variable was specified and predictions were made for that variable to compare prediction performance. In this performance comparison, the error index was the RMSE value, and the prediction model was set to KNN regression for the efficiency of the experiment.

As a result of the experiment, it was found that the prediction performance of the prediction model learned by utilizing the present invention was excellent for most variables. For example, in the case of the disk read data amount variable, the performance of the prediction model learned with variables selected based on correlation was 0.00641 better than the prediction model utilizing the present invention, but the prediction model of the present invention showed the best performance in other variables. Specifically, it showed 21% better performance than the prediction model with the worst performance in the case of CPU usage rate, 45% better performance in the case of memory usage rate, 15% better performance in the case of disk write data amount, 20% better performance in the case of network inflow, and 16% better performance in the case of network outflow.

As a result of the experiment, when the present invention showed excellent effects, CPU steal and CPU iowait were selected as representative variables for the target variables of CPU utilization. This is because, in terms of CPU utilization prediction performance, if values such as CPU steal or CPU iowait increase, it becomes difficult to perform other processes on the same physical machine or to distribute CPU resources to multiple virtual machines, thus affecting the overall CPU utilization.

In addition, variables related to file system usage and disk writing were selected as representative variables for the target variables of memory usage. This is because, in terms of memory usage prediction performance, many of the virtual machines being analyzed are related to data pipelines, so when values such as file system usage or disk writing increase, requests to store or query databases related to large amounts of data are processed, which affects memory usage.

In this way, the present invention can also be utilized to acquire additional cloud domain knowledge based on selected representative variables.

Below, a timing determination device (120) that recommends a re-training time for a learning model trained on a cloud server will be described in detail.

Figure 7 is a configuration diagram of a timing determination device (120) of a model management system according to one embodiment of the present invention.

A timing determination device (120) according to one embodiment of the present invention may include a timing receiving module (121) that receives the motion information in a time-series manner, a seasonal module (122) that decomposes the motion information in a time-series manner and then calculates the seasonal intensity of the motion information according to a predetermined intensity analysis method, a pattern module (124) that analyzes the pattern of the motion information using a predetermined pattern analysis method, and a judgment module (125) that determines the re-learning timing of the machine learning model performed in the cloud server using a predetermined decision method according to the magnitude of the seasonal intensity.

In addition, the timing determination device (120) may further include a preprocessing module (123) that performs a preprocessing process for reducing the dimension of motion information, a prediction module (126) that predicts future motion information, and a storage module (127) that stores information necessary for implementing a timing determination method.

The timing receiving module (121) can receive motion information from a cloud server.

The motion information is information on time-series values for multiple variables, and a detailed description may be omitted to the extent that it overlaps with the above-described content.

The seasonal module (122) can receive operation information, which is information generated when the cloud server operates.

The operation information can be information about the resource usage of the physical server (cloud server) while the cloud server is in operation.

For example, the operation information may include the central processing unit (CPU) usage, auxiliary memory (Memory) usage, network usage, main memory (Disk) usage, and file system usage of the cloud server.

For example, the motion information may further include CPU wait time (iowait) usage.

Here, usage can mean the amount of resources actually used compared to the resources allocated to the cloud server, or the amount of resources actually used compared to the maximum available resources of the cloud server.

Here, usage can mean the actual iowait time observed compared to the allowable iowait time limit on the cloud server.

Here, each usage amount that constitutes the motion information can be defined as indicator data.

The above operation information can be stored in the storage module (127).

The seasonal module (122) can calculate the seasonal intensity of the motion information according to a predetermined intensity analysis method after decomposing the motion information into time series.

The seasonal module (122) can decompose all types of operation information (CPU usage, memory usage, network usage, disk read/write usage, file system usage, and iowait values) into time series and classify them into trend, seasonality, and noise (residual).

For example, the seasonal module (122) can perform time series decomposition using an additive or multiplicative method, but the present invention is not limited thereto, and a description of a specific algorithm for a time series decomposition method may be omitted within the scope of known technology.

The predetermined intensity analysis method may be a method of analysis based on the seasonality of the above motion information and the noise of the above motion information.

Specifically, the above-determined strength analysis method may be a method following mathematical expression 5.

----------------------- [Mathematical Formula 5]

Here, is the intensity of seasonality of indicator data, is the seasonality of indicator data), is the noise of the indicator data, is the variance of the seasonality and noise sum of the indicator data, can mean the variance of noise in indicator data.

Additionally, the unit of seasonality can be a cycle.

The intensity of seasonality can be expressed as a value between 0 and 1.

The seasonal module (122) calculates the intensity of seasonality for each indicator data in the motion information, and by averaging the intensity of seasonality for each indicator data, it can calculate the intensity of seasonality for the motion information for the cloud server.

The seasonal module (122) can transmit the calculated seasonal intensity to the judgment module (125) and/or the storage module (127).

The storage module (127) can store the seasonal intensity produced by the seasonal module (122).

The preprocessing module (123) can perform preprocessing to reduce the dimension of the above motion information.

For example, the preprocessing module (123) can reduce the dimension of motion information by utilizing techniques such as AE (AutoEncoder), PCA, SVD, and NMF.

The preprocessing module (123) can reduce the dimension of each of the various indicator data constituting the motion information.

Due to this, reduced information can be produced for each indicator data (CPU usage, memory usage, network usage, disk usage, file system usage, iowait value).

The reduced information produced by the preprocessing module (123) can be stored in the storage module (127) in correspondence with the basic motion information.

The reduced information produced by the preprocessing module (123) can be transmitted to the pattern module (124) and/or the storage module (127).

The pattern module (124) can analyze the pattern of the above motion information using a pre-determined pattern analysis method.

Specifically, the pattern module (124) can analyze the pattern of the reduced-dimensional measurement information of the motion information using a pre-determined pattern analysis method.

Here, the predetermined pattern analysis method may be a method of analyzing a pattern based on the relationship between each of the reduced-dimensional indicator data constituting the reduced information.

As a specific example, the predetermined pattern analysis method may be a method of analyzing patterns as size ratios between each dimensionally reduced indicator data constituting the reduced information.

As a concrete example, if CPU usage is 30%, Memory usage is 25%, Network usage is 60%, Disk usage is 20%, Filesystem usage is 30%, and iowait usage is 15% (iowait value in operation information compared to pre-determined limit iowait time), the pattern of the operation information can be 1: 0.833 : 2 : 0.667 : 1 : 0.5.

The pattern of motion information produced by the pattern module (124) can be transmitted to the storage module (127) and/or the judgment module (125).

The prediction module (126) can predict the future minimum and maximum points in time for the amount of resources used by the cloud server.

Here, the lowest point can mean the point in time when the minimum amount of motion information (average of various indicator data) occurs in one cycle.

Additionally, the peak point can mean the point in time when the maximum amount of motion information (average of various indicator data) occurs in one cycle.

The prediction module (126) collects past motion data by calculating backwards from the current point in time, which is the point in time at which re-learning is determined from the storage module (127), and then determines which point in time the current point in time is in the cycle, and then predicts when the lowest or highest point in time will arrive in the future based on seasonality (cycle).

For example, if the seasonality is one year, the lowest point in the one-year cycle is four months from the time period, the highest point is eight months from the time period, and the current point is two months from the time period in the cycle, the prediction module (126) can predict that the lowest point will occur two months from the current point and the highest point will occur six months from the current point.

The prediction module (126) can transmit the prediction result to the judgment module (125).

The prediction module (126) can predict the re-learning time for the detection model.

For this purpose, when the detection model was machine-learned in the past, the amount of input data and the time required can be stored in the storage module (127).

The prediction module (126) can receive the amount of newly input data for re-learning from the cloud server, and can receive the time taken for the past learning model to learn and the amount of input data from the storage module (127) or the cloud server.

For example, the prediction module (126) can predict the re-learning time required for re-learning as a proportional relationship that the learning time increases as the amount of data increases and transmit the result to the judgment module (125).

The judgment module (125) can determine the re-learning period of a learning model (e.g., a future prediction model, an anomaly detection model) operated in a cloud server according to the magnitude of the intensity of seasonality of the cloud server.

This is because, in the case of cloud servers, quick judgment and response are required, so if machine learning is performed when there are many processes running on the cloud server, the cloud server's resources may become overloaded.

This may cause problems in the cloud server's functionality.

The predetermined decision method may be a method of determining the re-learning timing of the learning model by considering the seasonality of the motion information or the pattern of the motion information.

The predetermined decision method may include a first decision method for determining a re-learning time by considering the seasonality of the motion information when the seasonality intensity of the motion information is equal to or greater than a first reference value, and a second decision method for determining a re-learning time by considering the pattern of the motion information when the seasonality intensity of the motion information is lower than a second reference value that is lower than the first reference value.

In addition, the above-determined decision method may further include a third decision method for determining a re-learning period by considering the seasonality of the motion information or considering the pattern of the motion information depending on whether a predetermined intermediate condition is satisfied when the seasonality intensity of the motion information is greater than or equal to the second reference value and less than the first reference value.

For example, the first criterion may be 0.7, and the second criterion may be 0.4.

However, without limitation thereto, the specific numerical values of the first reference value and the second reference value can be varied in various ways at a level that is obvious to a person skilled in the art.

As a specific example, the first decision method can determine the future minimum point as the re-learning time of the learning model if the median time, which is a predetermined ratio of the time between the minimum point when the motion information is least generated and the maximum point when the motion information is most generated within the cycle of the motion information, is longer than the re-learning time of the detection model.

For example, the given ratio could be 50%.

However, without limitation to this, the specific numerical value of the given ratio can be varied in various ways at a level that is obvious to a person skilled in the art.

For example, if the time between the lowest and highest points is 48 hours, the intermediate time may be 24 hours.

This may be to prevent resource overload as retraining of the learning model may end before the peak point is reached.

As a specific example, the first determination method may be a method of determining a future point in time between the highest point and the lowest point as the re-learning time of the learning model while the motion information is decreasing, if the intermediate time, which is a predetermined ratio of the time from the lowest point, which is the point in time when the motion information is least generated, to the highest point, which is the point in time when the motion information is most generated, within the cycle of the motion information is less than the re-learning time.

For example, the given ratio could be 50%.

However, without limitation to this, the specific numerical value of the given ratio can be varied in various ways at a level that is obvious to a person skilled in the art.

This may be to prevent resource overload by ensuring that retraining of the learning model is not terminated before the peak point is reached.

Here, a future point in time between the highest point and the lowest point while the motion information is decreasing may mean a point in time before a predetermined percentage of time between the highest point and the lowest point while the motion information is decreasing from the lowest point.

The second decision method may be a method of determining the re-learning time of the learning model at the point in time when a pattern of motion information similar to a reference motion pattern, which is a pattern of motion information when the learning model was learning in the past, is confirmed to be higher than a predetermined similarity value, which is a maximum similarity criterion.

At this time, the second decision method may need to be performed with the same learning model as the learning model that determines whether or not to relearn. For example, when the timing of relearning judgment of a future prediction model needs to be determined, the timing of relearning may be determined based on a reference motion pattern, which is a pattern of motion information when the future prediction model was learned in the past.

The judgment module (125) can determine the highest similarity criterion based on the similarity value of the motion information pattern that is most similar to the above-mentioned standard motion pattern.

As a specific example, the judgment module (125) can determine the highest similarity criterion as the similarity value having the motion information pattern most similar to the reference motion pattern in the cloud server during a predetermined period of time (for example, 1 cycle) from the present time.

The judgment module (125) can calculate a similarity value based on the distance of the pattern, and the shorter the distance, the higher the similarity value.

For example, if there is a pattern of 1:2:3:4:5:6 and a pattern of 1:3:3:4:5:7, the distance between the two patterns can be calculated as (1-1)^2+(2-3)^2+(3-3)^2+(4-4)^2+(5-5)^2+(6-7)^2, which produces 2.

For example, a similarity value according to distance may be specified in the judgment module (125).

For example, if the distance is 3, the similarity value may be 90.

However, without limitation thereto, the specific values of the specified similarity values according to the distance can be varied in various ways at a level that is obvious to a person skilled in the art.

The standard motion pattern can be generated by analyzing the pattern by the preprocessing module (123) and the pattern module (124) based on the motion information collected when learning occurs, and then stored in the storage module (127).

For example, under the second decision method, if the most similar value to the reference motion pattern among the patterns of motion information that occurred during a predetermined period of time in the past is 98, the highest similarity criterion may be the similarity value 98.

The third decision method may be a method of determining the re-learning period of the detection model by considering the seasonality of the motion information or considering the pattern of the motion information depending on whether a predetermined intermediate condition is satisfied when the seasonality intensity of the motion information is greater than or equal to the second reference value and less than the first reference value.

Here, the above-determined intermediate condition may be a condition in which there exists motion information having a pattern similar to the reference motion pattern by a predetermined similarity value or more than a minimum similarity criterion during a predetermined past period.

For example, a predetermined past period could be a given amount of time in the past (e.g., one period) from the present time.

However, without limitation thereto, the above-determined past period can be varied in various ways at a level that is obvious to a person skilled in the art.

For example, the minimum similarity criterion could be a similarity value of 85.

For example, the minimum similarity criterion may be less than the maximum similarity criterion.

However, without limitation to this, the specific value of the minimum similarity criterion can be varied in various ways at a level that is obvious to a person skilled in the art.

A detailed description of the third decision method will be provided later.

The judgment module (125) can determine the re-learning point of the detection model and transmit it to the cloud server.

The judgment module (125) can determine the re-learning point of the learning model and transmit it to the interface device.

The storage module (127) can store information received from other modules in correspondence with the underlying operation information.

Additionally, the storage module (127) can store the motion information together with the time at which the motion information occurred or the time at which the seasonal module (122), preprocessing module (123) or storage module (127) received the motion information.

The storage module (127) can store all information necessary for implementing the timing determination method.

Hereinafter, a timing determination method according to one embodiment of the present invention will be described in detail.

Figure 8 is a flowchart of a timing determination method of a timing determination device according to one embodiment of the present invention.

Referring to FIG. 8, a timing determination method according to an embodiment of the present invention may include a step of receiving motion information from a cloud server by a timing receiving module (121), a step of analyzing the motion information in time series by a seasonal module (122) to calculate seasonality intensity of the motion information, a step of analyzing a pattern of the motion information according to a predetermined pattern analysis method by a pattern module (124), and a step of determining a re-learning time of a learning model performed in a cloud server according to a predetermined decision method according to the magnitude of the seasonality intensity by a judgment module (125).

Below, detailed explanations may be omitted to the extent that they overlap with the above-mentioned contents.

Figure 9 is an operational diagram of a timing determination device according to one embodiment of the present invention.

Referring to Fig. 9, a timing receiving module (121) can receive operation information from a cloud server.

The motion information is transmitted to the account module and the preprocessing module (123), respectively. The seasonal module (122) calculates the seasonal intensity of the motion information according to a predetermined intensity analysis method and transmits it to the judgment module (125), and the preprocessing module (123) can reduce the dimension of the motion information and transmit it to the pattern module (124).

The pattern module (124) can analyze the pattern of the above motion information using a pre-determined pattern analysis method and transmit it to the judgment module (125).

The judgment module (125) can select the first to third decision methods according to the intensity of seasonality to determine the re-learning time of the learning model.

The re-learning time determined by the judgment module (125) and the basis for the judgment are transmitted to the interface device, and the interface device can generate a user interface that allows the manager (M10) to examine the re-learning time and the basis for the judgment and transmit it to the manager (M10).

FIG. 10 is a graph for explaining a first decision method executed by a judgment module of a timing decision device according to one embodiment of the present invention.

Referring to FIG. 7 and FIG. 10, the lowest point (X11) may mean a point in time when motion information is least generated within a cycle of collected motion information, and the highest point (X12) may mean a point in time when motion information is most generated within a cycle of collected motion information.

The intermediate time (T11) can mean a certain percentage of the time from the lowest point (X11) to the highest point (X12).

If the intermediate time is longer than the re-learning time predicted by the prediction module (126), the judgment module (125) can determine the lowest time point (X14) that first appears in the future based on the current time point (X13) as the re-learning time.

In contrast, if the intermediate time is less than the re-learning time predicted by the prediction module (126), the judgment module (125) may determine the re-learning time as a time point (X16) between the highest time point (X15) and the lowest time point (X14) that shows a decreasing trend that will first appear in the future based on the current time point (X13).

In other words, if the intermediate time is less than the re-learning time predicted by the prediction module (126), the judgment module (125) may determine the re-learning time as a time point (X16) that is a predetermined percentage of the time between the highest time point (X15) and the lowest time point (X14) that show a decreasing trend that first appears in the future, based on the current time point (X13).

FIG. 11 and FIG. 12 are drawings for explaining a third decision method executed by a judgment module of a timing decision device according to one embodiment of the present invention.

Referring to FIG. 8 and FIG. 11, the third decision method may be a method of determining the future lowest point as the re-learning time of the learning model by considering the seasonality of the motion information when the above-determined intermediate condition is not satisfied. (Third point-in-time judgment method)

The method for considering the seasonality of the above motion information may be the same as the first decision method, and a detailed description thereof may be omitted to the extent that it overlaps with the above-described content.

Here, the above-determined intermediate condition may be a condition in which there exists motion information having a pattern similar to the reference motion pattern by a predetermined similarity value or more than a minimum similarity criterion during a predetermined past period.

The third decision method may be a method of determining the re-learning timing of the learning model in a different way depending on the number of motion information satisfying the above-determined intermediate conditions.

If the number of motion information satisfying the above-determined intermediate conditions is singular, the third decision method may be a method of determining the re-learning time point by the first time point judgment method.

The first point-in-time judgment method may be a method of determining a re-learning time at which a point in time when motion information having a pattern similar to the reference motion pattern by the set similarity value is confirmed by setting a similarity value between the pattern of motion information satisfying a predetermined intermediate condition and a reference motion pattern is confirmed.

The third decision method may be a method of determining the re-learning time of the detection model as the time point at which motion information having a similarity value with the reference motion pattern identical to the past motion information satisfying the predetermined intermediate condition, which is the lowest point at which the motion information is least generated in one cycle during the predetermined past period, is confirmed, when the number of motion information satisfying the above-described intermediate condition is greater than or equal to the first number and less than the second number greater than the first number. (Second time point judgment method)

For example, the first number could be 2, the second number could be 5, etc.

However, without limitation thereto, the specific values of the first number and the second number can be varied in various ways at a level that is obvious to a person skilled in the art.

Referring to FIG. 7 and FIG. 12(a), there can be a total of four pieces of motion information (W11, W12, W13, W14) that satisfy predetermined intermediate conditions during a predetermined past period (A11) based on the present time (Q11).

Additionally, the lowest point (S11, S12) may appear twice during a predetermined past period (A11).

Under the third decision method, the judgment module (125) can designate the similarity value between the pattern of the motion information (W12) closest to the lowest point and the reference motion pattern as the set similarity value, and determine the time point at which motion information having a pattern similar to the reference motion pattern by the set similarity value is confirmed as the re-learning time.

As a specific example, if the set similarity value is 95, the judgment module (125) can determine the re-learning period as the point in time when motion information having a pattern similar to the reference motion pattern by a similarity value of 95 is confirmed.

The third decision method may be a method of determining the re-learning time of the detection model at a time when motion information having a similar value to the reference motion pattern that is the same as the motion information of the past that satisfies the predetermined intermediate condition that is the furthest from the time at which the abnormal motion was detected during the predetermined past period is confirmed, when the number of motion information satisfying the predetermined intermediate condition is greater than or equal to the second number. (Second time point judgment method)

Referring to FIG. 7 and FIG. 12(a), there may be a total of 7 pieces of motion information (W11, W12, W13, W14, W15, W16, W17) that satisfy predetermined intermediate conditions during a predetermined past period (A11) based on the present time (Q11).

Additionally, during the predetermined past period (A11), the anomaly detection model can detect a total of two anomalies (F11, F12).

Under the third decision method, the decision module (125) can designate the similarity value between the pattern of the motion information (W11) that is furthest from the time point (F11, F12) at which the abnormality sign is determined and the reference motion pattern as the set similarity value, and determine the time point at which motion information having a pattern similar to the reference motion pattern by the set similarity value is confirmed as the re-learning time.

As a specific example, if the set similarity value is 95, the judgment module (125) can determine the re-learning period as the point in time when motion information having a pattern similar to the reference motion pattern by a similarity value of 95 is confirmed.

This may be because measures taken in response to abnormal symptoms should be given priority.

Accordingly, a detailed description thereof may be omitted to the extent that it overlaps with the above-mentioned contents.

Unlike what was described above, the learning model is open source and can be supplied from an external server.

In order to more clearly express the technical idea of the present invention, the attached drawings briefly express or omit components that are not related to or have little to do with the technical idea of the present invention.

Although the configuration and features of the present invention have been described above based on embodiments according to the present invention, the present invention is not limited thereto, and it will be apparent to those skilled in the art that various changes or modifications can be made within the spirit and scope of the present invention, and therefore it is made clear that such changes or modifications fall within the scope of the appended patent claims.

100 : Model Management System 110 : Variable Selection Device
120 : Timing Device

Claims (10)

A variable selection device that determines a representative variable, which is a variable representing the characteristics of the cloud server, based on operation information, which is information about each of a plurality of variables generated while the cloud server is operating;
The above variable selection device is,
The cloud server is provided with a selection receiving module that receives operation information, which is information about each of a plurality of variables, as information generated while the cloud server is operating, a preprocessing module that preprocesses the operation information received by the selection receiving module according to a pre-determined preprocessing method, a detection module that analyzes causality between the target variable, which is a pre-determined variable, and the comparison variable, which is a variable excluding the target variable, according to a pre-determined judgment method based on the preprocessed operation information, and a decision module that determines the comparison variable and the target variable having causality satisfying a pre-determined causality condition as the representative variable.
The above pre-determined preprocessing method is,
Including a method for removing the above operation information for a variable whose correlation coefficient with the above target variable is greater than a predetermined standard.
Model management system.
In the first paragraph,
The above predetermined judgment method is,
A method of determining causality between two variables using the Granger causality test.
Model management system.
In the first paragraph,
It further includes a search module that distinguishes and analyzes the operation information of the target variable for each predetermined time range and detects whether an event occurs in the operation information of the target variable;
The above detection module,
The operation information of the target variable in which the event occurred and the operation information of the comparison variable that occurred in the same time range are compared with each other according to the predetermined judgment method to analyze causality.
Model management system.
In the third paragraph,
The above pre-determined causal conditions are,
The condition in which the above pre-determined time range in which causality is recognized is the largest,
Model management system.
In the first paragraph,
The above pre-determined preprocessing method is,
A method for removing the above motion information having a categorical or variance below a predetermined standard,
Model management system.
In the first paragraph,
The above pre-determined preprocessing method is,
A method for correcting the time error between indicator data indicating the above-mentioned operation information of each variable.
Model management system.
A model management method for managing a model implemented by a model management system and operated on a cloud server,
A step of determining a representative variable, which is a variable representing the characteristics of the cloud server, by a variable selection device;
The step in which the above representative variables are determined is:
A step of receiving operation information, which is information generated when the cloud server is operated by a selection receiving module, each of which comprises a plurality of variables - the variables including a target variable, which is a predetermined variable, and a comparison variable, which is a variable other than the target variable;
A step in which the motion information received by the selection receiving module is preprocessed according to a predetermined preprocessing method by the preprocessing module;
A step in which the operation information of the target variable is distinguished and analyzed by the search module for each predetermined time range, and whether an event occurs is detected from the operation information of the target variable;
A step in which the operation information of the target variable in which an event occurred and the operation information of the comparison variable that occurred in the same time range are compared with each other according to a predetermined judgment method by the detection module to analyze causality; and
A step comprising: a step in which the comparison variable and the target variable, which have a causality satisfying a pre-determined causality condition, are determined as the representative variable by a decision module;
The above pre-determined preprocessing method is,
Including a method for removing the above operation information for a variable whose correlation coefficient with the above target variable is greater than a predetermined standard.
How to manage models.
In the first paragraph,
Further comprising a timing determination device for determining a timing for re-learning the machine learning model based on the above-mentioned operation information for the above-mentioned representative variable;
Model management system.
In Article 8,
The above timing device is,
It includes a time receiving module that receives the above motion information in a time series manner, a seasonal module that calculates the seasonal intensity of the motion information according to a predetermined intensity analysis method after time series decomposition of the motion information, a pattern module that analyzes the pattern of the motion information using a predetermined pattern analysis method, and a judgment module that determines the re-learning time of the machine learning model performed in the cloud server using a predetermined decision method according to the size of the seasonal intensity.
The above predetermined decision method is,
A method for determining the re-learning period by considering the seasonality of the above motion information or the pattern of the above motion information.
Model management system.
In Article 9,
The above predetermined decision method is,
A first determination method for determining a re-learning time by considering the seasonality of the motion information when the seasonality intensity of the motion information is greater than or equal to a first reference value, and a second determination method for determining a re-learning time by considering the pattern of the motion information when the seasonality intensity of the motion information is less than a second reference value that is lower than the first reference value.
Model management system.

Images