KR102473637B1 - Apparatus and method for managing trouble using big data of 5G distributed cloud system - Google Patents

Abstract

The present invention relates to an apparatus and method for managing failures by analyzing logs of big data in a 5G distributed cloud system. An apparatus for failure management according to the present invention includes a message conversion unit for converting a log of big data generated in the infrastructure of a cloud system into a message related to failure; a failure detection unit for detecting failures by applying a rule engine to the converted message; In order to display each detected failure on a screen, a failure graphic providing unit that generates a failure-occurring infrastructure object as topology-based graphic information and provides it to an operator terminal; and a failure recovery unit generating failure recovery information of the detected failure and providing it to an operator terminal.

Description

Apparatus and method for managing trouble using big data of 5G distributed cloud system}

The present invention, as a failure management technology, relates to an apparatus and method for providing failure management accompanied by data collection and analysis, failure expression and recovery using big data generated in a 5G distributed cloud system.

1 is a hierarchical structure of a cloud infrastructure constituting a conventional distributed cloud system 100 for 5G service. The first hardware layer 110 is composed of various types of physical devices such as servers, storages, and networks (switches). The second OS layer 130 is composed of software (eg, Linux) that operates and manages lower hardware. The third virtualization layer 150 divides physical resources into logical resources and dynamically allocates computing, networking, and storage resources when a customer requests. The fourth cloud layer 170 is a platform that serves to collectively control all resources of the upper three layers 110 to 150, and performs various commands requested by the upper layers 110 to 150 and provides responses thereto. . In addition, the cloud layer 170 provides cloud services to subscribers by executing applications such as services and products defined in the service and automation layer 180. Finally, the management layer 190 on the right side performs functions such as resource management, configuration and control, failure monitoring and recovery, etc. required to efficiently operate and manage the cloud system 100 having such a multi-layer structure.

In the infrastructure hierarchical structure of the cloud system 100, a failure occurring in any one layer affects upper and lower layers. In other words, it is necessary to figure out which layer a failure occurs, which upper layer or lower layer is affected by the failure, and which layer should be accessed to recover from a failure. need.

FIG. 2 is a schematic configuration diagram of a cloud platform 200 that is software of a cloud layer 170 in the distributed cloud system 100 of FIG. 1 . The cloud platform 200, which is core software constituting the distributed cloud system 100, uses a distributed architecture in which several independently developed subsystems 210 to 270 gather to form one cloud. In the case of the open stack cloud platform 200, which is used most recently, an architecture composed of seven subsystems 210 to 270 is the smallest unit constituting the cloud.

The cloud platform 200 includes a portal subsystem 210 through which customers access the cloud through the web, a virtual computing subsystem 220 that provides computing resources requested by customers, and a virtual networking subsystem that provides network resources. 230, a virtual storage subsystem 240 providing storage resources, an image subsystem 250 providing image (operating system) resources, an authentication management subsystem 260 providing security services, and an object type storage service. It is composed of an object storage subsystem 270 that provides.

When a failure occurs in the cloud platform 200 of the distributed architecture composed of these multiple subsystems 210 to 270, it is necessary to track which subsystem 210 to 270 caused the failure, and the effect of the failure on the surrounding subsystems. and analysis of the impact on other infrastructure layers is needed. Therefore, it takes a long time to monitor the operation of the entire cloud system, analyze the cause of failure, and recover, unless it is a skilled IT professional for a long period of time, which can significantly reduce the reliability of the system, so improvement measures are needed. If it is not for a long period of skilled IT experts, it takes a long time to monitor the operation of the entire cloud system, analyze the cause of failure, and recover, which can greatly reduce the reliability of the system, so improvement measures are needed.

FIG. 3 is a schematic flowchart of a conventional operation management process of the distributed cloud system 100 of FIG. 1 . In FIG. 3, the process flow of conventional operations management has the processes of collection 310, filter 330, presentation 350, and failover 370. In the collection 310 process, the cloud system 100 uses numerical information such as central processing unit (CPU) load rate, memory utilization rate, and hard disk utilization rate or important processes of the cloud platform 200 to monitor the failure state of the entire cloud. It collects structured data having a certain format that can be intuitively understood, such as operating status of services and services. In the processing of the filter 330, the cloud system 100 compares the collected structured data based on a specific reference value or filters based on a simple criterion for determining whether the status is in operation or not, and determines whether or not there is a failure. In the presentation 350 process, the failure event found by filtering is visually expressed on a table screen of a graphical environment such as the web or provided with audible information to notify the operator.

In the process of failure recovery 370, the operator who has identified the failure event restores the failure by resolving the system error, relying on standard documentation of failure handling or his own capabilities. This conventional cloud operation management method has the following three problems.

The first problem is simple collection and filtering of state information in order to determine whether the cloud platform 200 composed of various cloud infrastructure layers 110 to 190 as shown in FIG. 1 and subsystems 210 to 270 as shown in FIG. 2 has a failure. cannot detect logical errors occurring inside the system. In the case of a logical error, no failure information is displayed in the monitoring system, even though a VoC is actually received from the customer. Therefore, when an operator searches the status, configuration files, and log files of all systems based on the infrastructure layer (110 to 190) and subsystem (210 to 280) to identify and recover from failures, the number of operators depends on the skill of the operator. From hours to minutes this failure recovery time can take.

The second problem is that even if a failure occurs and is displayed on the web screen of the operation management system, it is expressed in a simple table GUI, so it is difficult to determine where the failure event occurred in the complex cloud platform as shown in FIG. 3, and its location Even if it is understood, it is difficult to determine what impact and whether the customer is affected by failure at the level of the entire infrastructure layer (110 to 190) and subsystem (210 to 270).

The third problem is that most of the log files of the currently occurring cloud platform contain program error information from the developer's point of view, so it is difficult for an operator without expertise in software and IT to understand what the message means and take action on it. In order to solve these problems, it is urgent to educate and nurture experts in the fields of IT, Network, and Software, but this has the disadvantage of taking a long time and cost.

Korea Patent No. 10-1636796 (2016.06.30.)

The present invention is to solve the above conventional problems, and collects logs of big data generated in a 5G distributed cloud system, analyzes them into messages, detects failures from the analyzed messages through a rule engine, and detects failures The purpose of the present invention is to inform the manager of the location and influence of the system and to provide a device and method for executing failure recovery.

Another purpose of the device is to convert big data of various log information including hardware, operating system, cloud platform, and network traffic in a cloud system into a message for detecting a failure.

Another object of the apparatus is to detect physical and logical failures through an analysis process of applying the converted message to a rule engine and comparing and/or conditions for detecting failures.

The device displays the detected failure in a graphical topology based on hardware, operating system, cloud platform and network traffic, immediately informs the operator of the location and impact of the failure, provides automatic processing results of failure recovery and guides for manual failure recovery. It serves a different purpose.

According to an aspect, an apparatus for managing failures using big data of a 5G distributed cloud system includes a message conversion unit that converts a log of big data generated in the infrastructure of the system into a message related to failure; a failure detection unit for detecting failures by applying a rule engine to the converted message; In order to display each detected failure on a screen, a failure graphic providing unit that generates a failure-occurring infrastructure object as topology-based graphic information and provides it to an operator terminal; and a failure recovery unit generating failure recovery information of the detected failure and providing it to the operator terminal.

The message conversion unit collects and stores logs for each object of the cloud infrastructure including objects of hardware, operating system, cloud platform, and network platform.

The message converting unit converts at least one log line in each log file into a message for detecting a failure.

The message conversion unit may include a server name of a server from which the logs are collected; File name in which the log is recorded; a line ID identifying a message converted by analyzing at least one log line in the log; a message key composed of the server name, the file name, and the line ID of the message; and log lines and converted messages are stored using a structure including a line buffer in which each log line analyzed as the message is stored.

The failure detection unit may include a rule group name including at least one filtering rule; the filtering rule; a filter condition processed in the filtering rule; and detecting a filtering key from a message using a structure of the rule engine including a filtering key detected from the message and to which the filter condition is applied, and generating an error when the filter condition applied to the detected filtering key is satisfied in the message. detected by

The failure detection unit may include: a rule name including hierarchical structure information of the infrastructure in which failure is detected; a rule description describing the failure; log file name in which the failure is detected; log rank, which defines the rank of the log; case defining the distinction between uppercase and lowercase letters in keywords in the log; at least one filtering rule including filter conditions of AND and OR and at least one keyword to which the filter condition is applied to detect the failure from the log line corresponding to the message; and recovery processing information of automatic recovery, operator manual recovery, and third expert recovery executed to recover the failure detected by satisfying the filtering rule. It uses the structure of the rule engine that includes a failure class defining the failure class of the failure.

The apparatus further includes a graphic DB in which objects of the infrastructure and hierarchical connection information between objects are stored for screen display, and the disability graphic providing unit displays objects and hierarchical connection information corresponding to the infrastructure requested by the operator terminal. It queries from the graphic DB, creates hierarchical connection information between the searched objects as graphic information, and provides the generated graphic information to the operator terminal.

The failure graphic providing unit transmits failure information to the operator terminal for screen display, if there is a failure in an object included in the graphic information displayed on the operator terminal.

The failure recovery unit provides the failure recovery information to the operator's terminal, and recovers the failure by executing automatic recovery process, operator manual recovery process, and third expert recovery process according to the operator's request.

The failure recovery unit transmits an automatic recovery command to the infrastructure in which a failure occurs, automatically executed or according to an operator's request, and provides a result of processing the command to an operator terminal.

The failure recovery unit, when the automatic recovery fails, transmits an operator manual recovery command to the infrastructure where the failure occurred at the request of the operator, provides a command-processed result to the operator terminal, and provides the automatic recovery or If the operator's manual recovery fails, failure information is notified to the expert terminal for recovery by a third expert.

According to another aspect, a method executed by an apparatus for managing a failure using big data of a 5G distributed cloud system includes converting a log of big data generated in the infrastructure of the system into a message related to the failure; detecting a failure by applying a rule engine to the converted message; Generating an infrastructure object with a failure as topology-based graphic information and providing it to an operator terminal for screen output of each detected failure; and generating failure recovery information of the detected failure and providing it to the operator terminal to recover the failure.

According to one aspect of the present invention, physical failures and logical failures occurring in the hierarchical structure of a cloud system and subsystems of a cloud platform may be detected through message analysis using big data logs and failure analysis based on a rule engine.

Detected failures are displayed on the graphical topology in which the objects constituting the infrastructure of the cloud system are placed, and the position and effect of occurrence of failures while the operator moves objects interconnected by the hierarchical structure of the cloud system and the subsystems of the cloud platform. can be immediately identified.

In the failure display screen, in order to recover the detected failure, guides for failure recovery may be simultaneously provided by dividing into automatic recovery execution, operator manual recovery execution, and third expert recovery execution.

The following drawings attached to this specification illustrate preferred embodiments of the present invention, and together with the detailed description of the present invention serve to further understand the technical idea of the present invention, the present invention is the details described in such drawings should not be construed as limited to
1 is a schematic hierarchical structure diagram of a conventional distributed cloud system for 5G service.
FIG. 2 is a schematic configuration diagram of a cloud platform that is software of the distributed cloud system of FIG. 1 .
3 is a schematic flowchart of a conventional operation management process of the distributed cloud system of FIG. 1;
4 is a schematic configuration diagram of a 5G distributed cloud system for managing failures according to an embodiment of the present invention.
5 is a schematic flowchart of a failure management service provided by the central manager of FIG. 4 .
6 is an exemplary diagram of code for collecting and storing logs in the central cloud of FIG. 4 .
7 is an exemplary diagram of a structure in which the central manager of FIG. 4 analyzes log lines of collected log files and converts them into messages.
FIG. 8 is an exemplary view of a rule engine format in which the central manager of FIG. 4 analyzes messages to detect failures.
9 is an exemplary view of a tree in which the central manager of FIG. 4 processes filtering rules of a rule engine.
10 is an exemplary view of a failure management screen created by the central manager of FIG. 4 based on the service topology of the cloud platform.
11 is a schematic flowchart of a method for managing failures in a 5G distributed cloud system according to an embodiment of the present invention.
12 is a detailed flowchart of the step of analyzing and converting logs into messages in FIG. 11 .
FIG. 13 is a detailed flowchart illustrating a step of generating and providing graphic information for failure management in FIG. 11 .
FIG. 14 is a detailed flowchart of the step of recovering from failure and providing processing results in FIG. 11 .

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. Prior to this, the terms or words used in this specification and claims should not be construed as being limited to ordinary or dictionary meanings, and the inventors use the concept of terms appropriately to describe their invention in the best way. It should be interpreted as a meaning and concept consistent with the technical idea of the present invention based on the principle that it can be defined. Therefore, the embodiments described in this specification and the configurations shown in the drawings are only one of the most preferred embodiments of the present invention and do not represent all of the technical spirit of the present invention. It should be understood that there may be equivalents and variations.

4 is a schematic configuration diagram of a 5G distributed cloud system 400 for managing failures according to an embodiment of the present invention.

A 5G distributed cloud system 400 providing a failure management service according to an embodiment of the present invention includes a central cloud 410 and a plurality of edge clouds 430 .

The central cloud 410 is a device of the present invention, and serves to centrally manage each edge cloud 430 installed in a wide-area office in each region. When the central cloud 410 is built as a server device having a failure management function, a failure management service may be provided to each edge cloud 430 .

The central manager 411 installed in the central cloud 410 performs data communication with the edge manager agent 431 of the edge cloud 430 to provide a failure management service. In addition, the central cloud 410 provides the failure management service by providing a web UI to an operator terminal. The failure management service includes log data analysis and message conversion, failure detection from the converted message, creation and provision of a topology graphic screen of the detected failure event, and screen GUI-based processing.

The edge cloud 430 includes an edge manager agent 431, an edge cloud controller 432, hardware 433, an operating system 434, a cloud platform 435, an NFV zone 436, and an IT zone 437. It is composed by

The edge manager agent 431 communicates with the central manager 411 to receive a failure management service according to the present invention from the central manager 411 in the process of providing cloud services to subscribers. In the failure management service, the edge manager agent 431 collects cloud logs and transmits them to the central cloud 410, receives and executes a failure recovery command from the central cloud 410 when a failure occurs, and sends the execution result to the central cloud 410. Report to cloud 410.

The edge cloud controller 432 controls hardware 433, operating system 434, and cloud platform 435 to provide cloud services. Hardware 433 includes servers, storage and network equipment. The operating system 434 is loaded into the hardware 433 and supports execution of the cloud platform 435 .

By executing the cloud platform 435, an NFV zone 436 and an IT zone 437 corresponding to a resource group providing infrastructure for 5G service are created and 5G cloud service is provided to subscribers. In the cloud service, the NFV zone 436 provides infrastructure for providing network communication services to subscribers, and the IT zone 437 provides infrastructure for providing application services to subscribers.

5 is a schematic flowchart of a failure management service provided by the central manager 411 of FIG. 4 . 6 is an exemplary diagram of code for log collection and storage in the central cloud 410 of FIG. 4 . 7 is an exemplary diagram of a structure in which the central manager 411 of FIG. 4 analyzes log lines of collected log files and converts them into messages. FIG. 8 is an exemplary view of a rule engine format in which the central manager 411 of FIG. 4 analyzes a message and detects a failure. FIG. 9 is an exemplary view of a tree in which the central manager 411 of FIG. 4 processes filtering rules of a rule engine. 10 is an exemplary diagram of a failure management screen created by the central manager 411 of FIG. 4 based on a service topology of a cloud platform to display detected failures. Hereinafter, it will be described with reference to FIGS. 5 to 10 .

Referring to FIG. 5 , the central manager 411 includes a message conversion unit 511 , an error detection unit 513 , an error graphic providing unit 515 and a failure recovery unit 517 .

Assuming that the central cloud 410 is a server device including a storage medium, a memory (eg, RAM), and a processor, each component 511 to 517 is installed in the storage medium in the form of a program, and the executed program is stored in the memory. is loaded into and processed by the processor.

Hereinafter, the operational management flow (① to ⑧) of the failure management service processed by each component 511 to 517 to solve the first to third problems described above will be described.

The message conversion unit 511 collects logs of big data generated from objects of each edge cloud 430 constituting the infrastructure of the distributed cloud system 400 (①), and uses the collected logs for failure analysis. It is converted into a message (②).

Here, the message converter 511 collects all log information of hardware, operating system, cloud platform and all network traffic information of network platform constituting each edge cloud 430 in real time and stores them in a storage-based large-capacity message queue. . The central cloud 41 is a storage server and may store logs collected by the message conversion unit 511 through a network.

Referring to FIG. 6 , for data collection and storage, generally widely used open sources may be used. Taking logstash, which is widely used recently, as an example, a config file in which an input file path 610 of a log file and an output bootstrap server 630 corresponding to a storage server are defined is created and then executed. When a config file is executed in each edge cloud 430 and a change occurs in an input log file of a monitoring target, the corresponding log is stored in the storage message queue of the central cloud 410 .

In the present invention, the use of a storage-based message queue rather than a generally used memory-based message queue means that a storage-based large-capacity message queue such as "apache kafka" is sufficient while ensuring appropriate speed when storing and reading big data. Because it provides extensibility. Memory-based message queues have better performance, but data may be lost in case of failure, and scalability problems may occur to apply large amounts of data.

Referring to FIG. 7 , the message conversion unit 511 of the central manager 411 analyzes log lines of collected log files, converts them into messages, and stores the converted messages through the message structure 700 .

The log file includes a plurality of log lines. In general, it is not easy to derive meaningful analysis results when reading and analyzing a single line from a log file that is displayed as a simple line. Therefore, a process of converting individual lines into a meaningful message format is required, and for this purpose, the message conversion unit 511 uses the following three message conversion rules.

The first message conversion rule is a case in which a log line and a message are converted 1:1, in which case one line itself is a message that has meaning in failure analysis. The second message conversion rule is a case where a log line and a message are converted n:1, and a case in which several lines form a message that has meaning in one failure analysis. The third message conversion rule is a case where log lines and messages are converted n:1 like the second message conversion rule, but the same message header ( 2017-05-15 15:48:43.768 2097 ERROR nova. compute.manager ).

The message structure 700 of the data structure shown in FIG. 7 includes fields of a server name (hostname), a log file name (logname), a line ID (line_id), a message key (msg_key), and a line buffer (msg_line_buf).

The server name is a server name that uniquely identifies a physical server constituting the infrastructure of the edge cloud 430 generating logs. For example, as "cnode-01.dj-01-01.kt." 'kt': business operator, 'dj-01-01': rack No. 01 of Cloud No. 01 installed in Daejeon Concentration Station, 'cnode-01' : Means the same server name as No. 01 computing server installed in a specific rack (01). The log file name corresponds to the log file name of the analysis target. The line ID is an identifier used to identify the same error message when analyzing a log line in a log file, and uses the header of the log format (date, time, level, module name) used in most software. This may vary depending on the type of file. That is, the line ID corresponds to the message ID. The message key is an identifier combining the server name, log file name, and line ID (hostname + logname + line_id) to uniquely identify the message of the line currently being analyzed in all log files generated in the entire edge cloud 430 ( identification key). The line buffer is a collection of log lines having the same message key, and when a message is completed, it corresponds to an aggregate in which each log line is stored.

Then, the message conversion unit 511 reads the log lines of each log file one by one from the DB of the storage server, applies the first to third message conversion rules to the log lines, and displays the message identified by the line ID in the message structure 700. And message conversion is processed by storing information such as log lines stored in the line buffer.

In FIG. 5, the failure detection unit 513 refers to the converted message using the message structure 700, applies a rule engine to the message (③), and detects failure by processing the rule engine (④). ).

Referring to FIG. 8 , a rule engine format applied by the failure detection unit 513 of the central manager 411 is illustrated. The format of the rule engine defines the structure 800 of the rule engine using a hierarchical structure consisting of [ filter_rule_group->filter_rule->filter_condition->filter_key ]. In the hierarchical structure of the rule engine, filter_rule_group is the first layer corresponding to the rule group name. filter_rule is a lower layer of the first layer and is a second layer corresponding to a filtering rule belonging to a rule group of the upper layer 1. filter_condition is a lower layer of the second layer, and is a third layer corresponding to a filter condition that is compared and processed in the filtering rule of the upper second layer. filter_key is a lower layer of the third layer, and is a fourth layer corresponding to a filtering key that is compared and processed by filter conditions of the upper third layer.

The rule engine structure 800 includes a rule name (name), a rule description (desc), a log file name (logfile), a log level (loglevel), a case, a filter condition (cond), and a keyword (keyword). It includes fields of a filtering rule (body), recovery process information (shooter), and failure level (fltlevel).

The rule name "cloud.openstack.system.quota_exceed" is a name that describes failure analysis rules of the rule engine, and is structured and easy to understand by using a hierarchical structure separated by '.'. The rule description (desc) briefly describes the failure detected by the rule engine so that the operator can easily understand. The rule description may be provided to the operator terminal 530 . The log file name (logfile) is the name of a log file in which a message to which the failure analysis process of the rule engine is applied is converted. That is, the failure detection unit 513 applies the filtering rule of the rule engine structure 800 having a log file name matching the log file name of the converted message to the message. The log level is a log level defined by an administrator. The highest log level indicates an important log for failure analysis. The case indicates whether or not case distinction is used when searching for a keyword from a message. When filtering messages by the rule engine, three items are used as basic conditions: log file, log level, and case.

In the structure 800 of the rule engine, the rule group name of the first layer is "openstack_expert_trouble_map". This first layer of rule groups are "cloud.openstack.system.quota_exceed" and "cloud.openstack.system.no_valid_host" It has two second-tier filtering rules named

Here, the second layer filtering rule “cloud.openstack.system.quota_exceed” has two third layer filter conditions defined as “01” and “02”. These filter conditions are sequentially applied, and only when the previously applied filter condition "01" is satisfied, the next filter condition "02" is applied. And the '01' filter condition uses two fourth-layer filtering keys of "exception" and "quota" as the case "ignore", and operates as an "and" filter condition. That is, when the rule engine of the error detection unit 513 detects the keywords "exception" and "quota" from the log message, it determines that the "01" filter condition is successful. If the keywords of "exception" and "quota" are not detected, the "01" filter condition is a failure, and processing of the '02' condition is skipped. If both of the filter conditions of “01” and “02” are successful, an error is detected through the message analysis process of the rule engine.

The recovery processing information (shooter) is address information of a program that is executed for recovery from a failure when a message to be analyzed satisfies a filtering rule of a filter condition. If a failure occurs without recovery process information, a third expert should be contacted to seek cooperation. The failure level (fltlevel) is a failure level determined by an administrator defining filtering rules. For example, failure grades may be classified into WR (Warning)/ER (Error)/CR (Critical).

The failure detection unit 513 analyzes and processes the message, reads log data such as all figures and states included in the message, applies various filtering rules defined by the administrator in real time to detect failure events, and outputs the results to the message queue. save to The filtering rules of this failure analysis make it a knowledge management system that can be continuously updated and added, making it an important operational management asset.

Referring to FIG. 9 , the filtering rules of the rule engine processed by the failure detection unit 513 are shown in a tree structure. Whenever the message conversion unit 511 generates a message through log file analysis, the failure detection unit 513 reads the generated message from the message queue. The failure detection unit 513 loads a filtering rule group of the rule engine structure 800 having a log file name matching the log file name of the message. The loaded filtering rule groups may be displayed in a tree structure. Here, if there is a change in the filtering rule group, it can be newly read and dynamically applied, and if there is no change, the initially loaded content can be used.

In the tree structure, the role group of the first layer becomes a root node and has a filtering rule of the second layer as a child node. A filtering rule node of the second layer becomes a parent node and has a filter condition of the third layer as a child node. In addition, the filter condition node of the third layer becomes a parent node and has the filtering key of the fourth layer as a child node. Tree traversal corresponds to the filtering process of the rule engine.

In the tree traversal, node 901 of the filtering rule group is visited. The node 901 of the filtering rule group has first and second filtering rule nodes 902 and 910 . First, it is visited from the first filtering rule node 902 and the first filtering rule is applied to the log message. The filtering rule node 902 has first and second filter condition nodes 903 and 907, which are applied one by one starting from the first filter condition node 903. A log message is determined by applying the 'and' or 'or' condition of the first filter condition node 903 to keywords of the first to third filtering key nodes 904, 905, and 906 as child nodes. That is, it is determined whether filter conditions for three keywords in the log message are successful. If the filtering process is successful in the first filter condition node 903, the filter condition of the second filter condition node 907 is applied to the keywords of the first and second filtering key nodes 908 and 909, which are child nodes, so that a log message is generated. judged If the filtering process is successful in both the first and second filter condition nodes 903 and 907, a failure is detected by the first filtering rule in the first filtering rule node 902. When the filtering process of the first filtering rule node 902 is completed, the filtering process of each node 911 to 918 is applied by visiting the second filtering rule node 910 .

5, the failure graphic providing unit 515 receives a request for a failure management screen from the operator terminal 530 (⑤), and provides topology-based graphic information for each object constituting the cloud infrastructure to display the requested screen. It creates (⑥), and provides graphic information for displaying the failure management screen to the operator terminal 530 (⑦).

Referring to FIG. 10 , the operator terminal 530 displays a failure management screen 1000 generated with 9 infrastructure objects and their connection information based on the topology of the cloud infrastructure.

The obstacle graphic providing unit 515 refers to a graphic DB in which objects of the cloud infrastructure and hierarchical connection information between objects are stored in order to generate a screen requested by an operator. Then, the obstacle graphic providing unit 51 generates hierarchical connection information and failure information between objects queried from the graphic DB as topology-based graphic information, and transmits the generated graphic information to the operator terminal 530 as screen display information. to provide. The topology-based graphic information is information indicating connections between constituent objects constituting the infrastructure hierarchical structure of the distributed cloud system 100 shown in FIG. 1 .

Here, the failure graphic provider 515 provides detailed diagrams of the infrastructure, platform, and customer service as graphic information so that the failure can accurately identify the position of the infrastructure object and easily analyze the cause. Then, the operator terminal 530 creates a failure management screen based on the GUI of the graphic information. On the failure management screen, summary information of each object is displayed, and an integrated operation environment is provided in which detailed logs and past history data related to failure events can be easily inquired.

After the failure management screen 1000 generated by the graphic information received from the operator terminal 530 is displayed, the operator selects an object displayed on the screen and transfers the upper and lower object screens to the failure graphic providing unit 515. Request is made and corresponding screen display information is received from the obstacle graphic providing unit 515, and a screen is generated and displayed. In addition, whenever a failure occurs, the failure graphic provider 515 generates failure information and screen display information indicating the failure and transmits them to the operator terminal 530 in real time. That is, whenever a failure is detected in an object displayed on the screen, it is displayed on the failure management screen of the operator terminal 530 in real time. Then, in the operator terminal 530, the operator requests detailed information and recovery information for the failure information displayed on the screen, and the operator terminal 530 receives the corresponding screen from the failure graphic providing unit 515 receiving the operator's request. Receives information and displays it on the screen.

In FIG. 5 , the failure recovery unit 517 generates recovery guide information of the detected failure and provides it to the operator terminal 530 (⑧). The operator issues a recovery command by referring to the received failure recovery guide information, and the failure recovery unit 517 transmits the recovery command to the edge manager agent 431 of the edge cloud 430 where the failure occurred. The failure recovery unit 517 receives a recovery process result from the edge manager agent 431 and transmits it to the operator terminal 530 .

Here, the failure recovery unit 517 may provide a description of the rules of the structure 800 and the data of the detailed log for the failure to the failure recovery-only window of the operator terminal 530 . Failure recovery processing may vary according to the administrator's failure recovery policy. For example, first automatic recovery, second operator manual recovery, and third third expert recovery may be sequentially performed. According to the defined failure recovery policy, if automatic recovery is possible, the failure recovery unit 517 communicates with the operator terminal 530 in both directions according to the procedure, performs an automatic recovery command, and the operator terminal 530 recovers the failure. display the result on the screen. If it is impossible to automatically recover the failure, the failure recovery guide is displayed step by step on the screen so that the operator can recover the failure himself. If the automatic recovery and operator recovery of the last failure occurred or if it is preset as an expert recovery, a third expert is notified of the recovery.

11 is a schematic flowchart of a method for managing failures in the 5G distributed cloud system 400 according to an embodiment of the present invention. For the method of failure management, the description described above with reference to FIGS. 5 to 10 may be used below.

The central cloud 410 collects logs of big data generated from each object constituting the infrastructure of the system 400 (S1111). When logs are collected in the storage server, the central cloud 410 analyzes the logs stored in the storage server and converts them into messages (S1113).

When a message converted by log analysis is generated, the central cloud 410 detects a failure by processing the filtering rules of the rule engine (S1121). It is possible to detect logical errors as well as physical errors of the system 400 as failures by means of messages converted from big data logs and filtering rules of the rule engine.

Thereafter, when the operator requests a failure management screen from the operator terminal 530, the central cloud 410 receives the operator's screen request (S1131), and the central cloud 410 converts the topology of the cloud infrastructure for the requested screen. Based on this, screen display information is generated, and generated failure information is added and provided to the operator terminal 530 (S1133). In addition, when a failure occurs in the system 400, the central cloud 410 provides failure information to the operator terminal 530 to display the failure information in real time (S1135).

Here, the operator terminal 530 generates and displays a failure management screen based on screen display information provided from the central cloud 410 . In this failure management screen, each infrastructure object constituting the system 400 and connection information between objects are displayed in a hierarchical association structure. That is, the operator can request and receive detailed failure information while moving to the upper, lower, and connection objects of each object on the screen.

When the operator requests recovery on the failure management screen, the central cloud 410 receives the operator's recovery request from the operator terminal 530 (S1137). The operator may request automatic recovery, request recovery by sending an operator's manual command, or request recovery by a third expert. When a recovery request is received, the central cloud 410 transmits a corresponding recovery command to the edge cloud 430 for recovery processing, and transmits the recovery processing result received from the edge cloud 430 to the operator terminal 530 ( S1139). If recovery of the third expert is requested, the central cloud 410 requests failure recovery by notifying failure information to the expert terminal.

FIG. 12 is a detailed flowchart of the step (S1113) of analyzing the log and converting it into a message in FIG. 11 .

First, the basic log files of the cloud system 400 can be classified into infrastructure logs providing logs related to hardware, system logs provided by the hardware operating system, and platform logs providing the most information related to the cloud. The central cloud 410 stores entire log lines from each log file stored in the storage server in a message queue for log analysis.

The central cloud 410 reads the log line from the message queue and determines whether a log line to be analyzed exists (S1211). If the log line exists, the central cloud 410 identifies the occurrence location of the log line using the server name (hostname) and the log file name (logname) (S1212), blanks for the log line, removal of unnecessary lines, etc. The same parsing process is performed (S1213), and it is determined whether a valid line ID (line_id) exists in the corresponding line (S1214).

If a valid line ID exists, if it is the first log line after starting the conversion program (S1215), the central cloud 410 initializes the item of the message structure 700 and adds the corresponding line to the line buffer (msg_line__buf) (S1225 ), the next line is read from the message queue (S1211).

If the log line does not have a valid line ID, it is a case where only additional contents are provided to the preceding log line without a log message header, so a line is added to the line buffer (msg_line_buf) created with the message key (msg_key) of the preceding line. And (S1224), the next line is read from the message queue (1211).

If the log line has a valid line ID and the same message key composed of the line ID already exists (S1216), the message key of the log line is added to the line buffer used as a key (S1224).

If a log line has a valid line ID and a message key composed of this line ID does not exist, it is determined that the message of the log lines using the previous message key is complete, and each line stored in the line buffer is combined to form a single message. It is completed, and the generated message is stored in the queue (log_message) (S1217). When one message is stored, as in step S1225, the message structure 700 is initialized and a corresponding line is added to the line buffer. Then, the next line is read from the message queue (log_line) (S1211).

Also, the same message group is usually created with several log lines in a short amount of time. Previously, one log line was read and a value was stored in the message structure 700, but if a new log line is not read for a long time, the contents of the message structure 700 may not be processed quickly and may fall into a waiting state for a long time. There is a problem. Accordingly, if the message queue of the log line is read three times in succession, but there is no log line (S1221), it is determined that the data stored in the previous message structure 700 is an already completed message, and the message completion and handle storage.

FIG. 13 is a detailed flowchart of generating and providing graphic information for failure management in FIG. 11 (S1133).

The operator terminal 530 accesses the central cloud 410 through the web and requests the central cloud 410 to display a failure management screen in order to monitor the cloud system 400 through the web access (1301).

Upon receiving the request, the central cloud 410 loads a class to automatically generate a graphic data structure suitable for the topology (infrastructure, platform, customer service) of the screen requested by the operator (1311). The central cloud 410 generates graphic objects necessary to express the graphic topology and link information to connect the objects (1312). When the graphic object and link information of the graphic topology is generated, the central cloud 410 determines the screen size based on the generated information and creates a screen size data structure in which the determined screen size information is stored (1313). That is, since the number of objects to be displayed and the connection links are different according to the failure management screen requested by the user, object information and link information to be displayed on the screen are queried from the DB, and the size of the screen is determined based on the queried information. .

When screen size information including screen ID and screen size (width/height) is stored in the screen size data structure, the central cloud 410 transfers arrangement information of graphic objects constituting the determined screen to object ID and title. , A screen object data structure composed of graphic object types, constraints to be displayed on the screen, etc. is created (S1314). Then, a link data structure is created in which screen link identifiers, link start identifiers, link end identifiers, arrow directions of links, and values of link line types of connection links for connecting these screen objects are stored (S1315). Here, the ID used in the screen size data structure, screen object data structure, and screen link data structure can be used as the same name in the failure information to express failure in the topology. The central cloud 410 transmits graphic information of screen data structures of screen sizes, screen objects, and screen links, respectively, to the operator terminal 530 (S1316).

Referring to the screen data structure with reference to FIG. 10, a screen is displayed based on the size of the screen size data structure in the failure guidance screen of FIG. 10, and 9 graphic object nodes are displayed according to the screen object data structure, Connection lines between 9 objects are displayed according to the screen link data structure.

The operator terminal 530 creates and displays a failure management screen using the screen data structure received from the central cloud 41 (S1321). The operator terminal 530 requests failure information to the central cloud 410 at predetermined intervals (S1322).

The central cloud 410 requested to provide failure information transmits failure information that can be displayed based on screen information in real time when failure information exists (S1331) (S1332). The operator terminal 530 displays the failure information received from the central cloud 410 on the failure management screen (S1341).

FIG. 14 is a detailed flowchart of the step S1139 of recovering a failure and providing a process result in FIG. 11 .

After the failure information display step (S1314) of FIG. 13, when the operator presses the failure recovery button to recover the failure displayed on the screen, the operator terminal 530 outputs a failure summary explanation on the screen to easily identify the failure, , requesting guidance of disaster recovery to the central cloud 510 .

Upon the operator's failure recovery request, the central cloud 410 receives a failure recovery guide request from the operator terminal 530 (S1411). Upon receiving the guidance request, the central cloud 410 refers to the field of the recovery processing information of the rule engine structure 800 for the requested failure (S1412). The referenced recovery processing information may be any one of automatic recovery, operator manual recovery, and third expert recovery. In the referenced recovery processing information, when automatic recovery is referred to (S1413), the central cloud 410 transmits an automatic recovery screen to the operator terminal 530 and requests the operator to select execution of automatic recovery (S1414), An automatic recovery execution request is received from the operator terminal 530 (S1415).

When the failover command is first initiated from the edge cloud 430 that is not in a connected state (S1421), the central cloud 410 accesses the edge cloud 430 (S1422), and the edge manager agent 431 automatically A recovery command is transmitted (S1423). When failure recovery is executed in the already connected edge cloud 430, the connection processing step (S1422) is omitted.

When an automatic recovery command is transmitted from the central cloud 410, the edge manager agent 431 receives and executes the automatic recovery command. The edge manager agent 431 transmits the execution result of the failover command to the central cloud 410 .

The central cloud 410 receives the execution result from the edge manager agent 431 and transmits it to the operator terminal 530 (S1424). Then, the operator terminal 530 displays the execution result received from the central cloud 410 on the screen.

If the execution result is successful (S1431), it is checked whether there is a next step of recovery from failure (S1432), and if there is a next task, the user returns to the step of selecting execution (S1415). If the next job does not exist, the recovery process for the failure is ended.

If the process result of the automatic recovery command is determined to be 3 consecutive failures (S1441), according to the failure recovery policy set by the administrator, the recovery processing information of the failure is replaced with the automatic recovery command that failed 3 times in a row. It is changed to the operator's manual recovery or the third expert recovery, and the recovery process of the failure returns to the above step S1413.

In addition, in the above step 1413, it is determined whether the failure rather than automatic recovery is an operator manual recovery (S1451). S1453). Here, the central cloud 410 provides an operator manual for failure recovery to the operator terminal 530, executes the operator's commands in the edge cloud 430, and transmits the execution result of the command to the operator terminal 530. do. Of course, if the operator's manual recovery fails, the failure recovery information of the failed failure can be changed to a third expert recovery.

In addition, if the failure recovery information of the failure is the third expert recovery, the central cloud 410 notifies the failure recovery with reference to the third expert information (S1461).

Although the present invention has been described with limited examples and drawings, the present invention is not limited thereto, and the technical idea of the present invention and claims to be described below are made by those skilled in the art to which the present invention belongs. Of course, various modifications and variations are possible within the equivalent range of the scope.

100: cloud system

Claims (22)

In the device for managing failures using big data of a 5G distributed cloud system,
Log lines with the same message key are added to the same line buffer by reading log lines one by one from the message queue storing the log of big data generated in the infrastructure of the system. a message converting unit for combining log lines stored in the line buffer and converting them into a message related to a single failure when there are none;
a failure detection unit for detecting failures by applying a rule engine to the converted message;
In order to display each detected failure on a screen, a failure graphic providing unit that generates a failure-occurring infrastructure object as topology-based graphic information and provides it to an operator terminal; and
Failure recovery unit for generating failure recovery information of the detected failure and providing it to the operator terminal
A device comprising a. According to claim 1,
The message conversion unit,
A device characterized in that for collecting and storing logs for each object of a cloud infrastructure including objects of hardware, operating system, cloud platform and network platform. According to claim 1,
The message conversion unit,
An apparatus characterized by converting at least one log line in each log file into a message for detecting a failure. According to claim 1,
The message conversion unit,
Server name of the server from which the log was collected;
File name in which the log is recorded;
a line ID identifying a message converted by analyzing at least one log line in the log;
a message key composed of the server name, the file name, and the line ID of the message; and
A line buffer in which each log line analyzed by the message is stored
Device characterized in that for storing the log line and the converted message using a structure comprising a. According to claim 1,
The failure detection unit,
a rule group name containing at least one filtering rule;
the filtering rule;
a filter condition processed in the filtering rule; and
A filtering key that is detected from the message and to which the filter condition is applied
Detecting a filtering key from a message using the structure of the rule engine including a rule engine, and detecting the occurrence of a failure when a filter condition applied to the detected filtering key is satisfied in the message. According to claim 1,
The failure detection unit,
a rule name including hierarchical structure information of the infrastructure in which failure is detected;
a rule description describing the failure;
log file name in which the failure is detected;
log rank, which defines the rank of the log;
case defining the distinction between uppercase and lowercase letters in keywords in the log;
at least one filtering rule including filter conditions of AND and OR and at least one keyword to which the filter condition is applied to detect the failure from the log line corresponding to the message; and
recovery process information of automatic recovery, operator manual recovery, and third-expert recovery executed to recover the failure detected by satisfying the filtering rule;
Disability class defining the class of disability of the above disability
An apparatus characterized in that using a structure of the rule engine comprising a. According to claim 1,
Further comprising a graphic DB in which objects of the infrastructure and hierarchical connection information between objects are stored for screen display,
The disability graphic providing unit inquires objects and hierarchical connection information corresponding to the infrastructure requested by the operator terminal from the graphic DB, generates hierarchical connection information between the searched objects as graphic information, and converts the generated graphic information to the graphics database. An apparatus characterized in that it is provided as an operator terminal. According to claim 1,
The disability graphic providing unit,
If an obstacle exists in an object included in the graphic information displayed on the operator terminal, the apparatus characterized in that for transmitting the failure information to the operator terminal for screen display. According to claim 1,
The failure recovery unit,
An apparatus characterized by providing the failure recovery information to an operator's terminal and restoring the failure by executing an automatic recovery process, an operator manual recovery process, and a third expert recovery process according to an operator's request. According to claim 1,
The failure recovery unit,
According to an automatic execution or an operator's request, an automatic recovery command is transmitted to the infrastructure in which a failure occurs, and a result of processing the command is provided to an operator terminal. According to claim 10,
The failure recovery unit,
When the automatic recovery fails, at the request of the operator, an operator manual recovery command is transmitted to the infrastructure where the failure occurred, and a result of the command processing is provided to the operator terminal,
and when the automatic recovery or the operator manual recovery fails, failure information is notified to an expert terminal for recovery by a third expert. A method executed by a device for managing failures using big data of a 5G distributed cloud system,
Log lines with the same message key are added to the same line buffer by reading log lines one by one from the message queue storing the log of big data generated in the infrastructure of the system. combining log lines stored in the line buffer when there are none, and converting them into a message related to one failure;
detecting a failure by applying a rule engine to the converted message;
Generating an infrastructure object with a failure as topology-based graphic information and providing it to an operator terminal for screen output of each detected failure; and
Restoring the failure by generating failure recovery information of the detected failure and providing it to the operator terminal
How to include. According to claim 12,
The step of converting the message into the
A method characterized in that the step of collecting and storing logs for each object of a cloud infrastructure including objects of hardware, operating system, cloud platform and network platform. According to claim 12,
The step of converting the message into the
and converting at least one log line in each log file into a message for detecting a failure. According to claim 12,
The step of converting the message into the
Server name of the server from which the log was collected;
File name in which the log is recorded;
a line ID identifying a message converted by analyzing at least one log line in the log;
a message key composed of the server name, the file name, and the line ID of the message; and
A line buffer in which each log line analyzed by the message is stored
characterized in that the step of storing the log line and the converted message using a structure including a. According to claim 12,
The step of detecting the failure is,
a rule group name containing at least one filtering rule;
the filtering rule;
a filter condition processed in the filtering rule; and
A filtering key that is detected from the message and to which the filter condition is applied
and detecting a filtering key from a message using a structure of the rule engine including a rule engine, and detecting that a failure occurs when a filter condition applied to the detected filtering key is satisfied in the message. According to claim 12,
The step of detecting the failure is,
a rule name including hierarchical structure information of the infrastructure in which failure is detected;
a rule description describing the failure;
log file name in which the failure is detected;
log rank, which defines the rank of the log;
case defining the distinction between uppercase and lowercase letters in keywords in the log;
at least one filtering rule including filter conditions of AND and OR and at least one keyword to which the filter condition is applied to detect the failure from the log line corresponding to the message; and
recovery process information of automatic recovery, operator manual recovery, and third-expert recovery executed to recover the failure detected by satisfying the filtering rule;
Disability class defining the class of disability of the above disability
The method characterized in that using the structure of the rule engine comprising a. According to claim 12,
The device includes a graphic DB in which objects of the infrastructure and hierarchical connection information between objects are stored for screen display,
In the providing step, an object corresponding to the operator's request and hierarchical connection information are inquired from the graphic DB, hierarchical connection information between the inquired objects is generated as graphic information, and the generated graphic information is converted to the operator terminal. A method characterized in that the step of providing as. According to claim 12,
The step of providing,
and transmitting failure information to the operator terminal for screen display, if an obstacle exists in an object included in the graphic information displayed on the operator terminal. According to claim 12,
The recovery step is
and providing the failure recovery information to an operator terminal and restoring the failure by executing an automatic recovery process, an operator manual recovery process, and a third expert recovery process according to an operator's request. According to claim 12,
The recovery step is
The method characterized in that the step of transmitting an automatic recovery command to the infrastructure in which a failure occurs according to automatic execution or an operator's request, and providing a result of processing the command to an operator terminal. According to claim 21,
The recovery step is
When the automatic recovery fails, transmitting an operator manual recovery command to the infrastructure where the failure occurs according to an operator's request, and providing a result of processing the command to the operator terminal; and
and notifying failure information to an expert terminal for recovery by a third expert when the automatic recovery or the operator manual recovery fails.

Images