KR102321205B1 - Apparatus for Providing AI Service and Driving Method Thereof - Google Patents

Abstract

The present invention relates to an AI service apparatus and a driving method thereof. According to an embodiment of the present invention, the AI service apparatus may comprise: a communication interface unit sequentially receiving video frames of a filmed image; and a control unit setting an event section of an object detection error based on a confidence score calculated relating to a designated object in each of the received video frames and allowing learning and reusing of an error detection objection of the video frames where the confidence score deviates a threshold in the set event section.

Description

Artificial intelligence service device and driving method thereof

The present invention relates to an artificial intelligence service apparatus and a method of driving the apparatus, and more particularly, to an artificial intelligence service apparatus for extracting meaningful data close to false positives or false positives in a post-processing process using, for example, a detector operating in the field, and an artificial intelligence service apparatus and the same It relates to a method of driving a device.

When artificial intelligence (AI) is divided into data standards, learning with data with correct answers is called supervised learning, and learning with general data without correct answers is called unsupervised learning. . Unsupervised learning requires a lot of data and the effect of learning is not that great. For this reason, modern AI is mostly supervised learning. As such, supervised learning-based deep learning models require trained data.

Usually, in industrial fields, detectors learned from data obtained from the site are used, but false positives or false positives occur. At this time, if false or false positive data is re-learned, the performance of the detector is improved, but it is difficult to obtain meaningful data. Of course, there are various methodologies related thereto in the prior art, but there is a problem that the development cost is high.

For example, in the related art, data that is meaningful for learning, ie, has a high probability of false positives or false positives, requires a high cost because a person observes the point in time and performs image backup or applies a methodology to extract the data.

Korean Patent Application Laid-Open No. 10-2021-0006502 (2021.01/18) Korean Patent Publication No. 10-2168558 (2020.10.15)

Website https://en.wikipedia.org/wiki/Active_learning Website https://inspaceai.github.io/2019/06/05/ActiveLearning_Introduc tion/

An object of the present invention is to provide an artificial intelligence service apparatus for extracting meaningful data close to false positives or false positives in a post-processing process using, for example, a detector operating in the field, and a method of driving the apparatus.

An artificial intelligence service apparatus according to an embodiment of the present invention includes a communication interface unit for sequentially receiving video frames of a captured image, and an object detection error ( false), and a control unit configured to reuse an erroneous detection object of a video frame in which the confidence score deviates from a threshold within the set event interval during learning.

The controller may determine the number of video frames for image analysis in order to set the event period.

The controller may determine whether there is an object detection error by comparing it with surrounding video frames to set the event period.

The controller may store the image of the erroneous detection object as supervised learning data and then reuse it during learning.

The control unit further stores a coordinate value in a video frame for the erroneous detection object by matching the image, and the coordinate value is calculated based on a coordinate value of a specified object corresponding to the erroneous detection object in a surrounding video frame. can

The control unit extracts the coordinate values of the specified object from video frames of a previous time and a subsequent time based on the video frame of the erroneous detection object, and calculates a median value of the extracted coordinate values of the erroneous detection object. Coordinate values can be calculated.

The controller may modify object information for the same object of the false detection object or change the state of a boundary box around the object according to reuse when learning the false detection object.

In addition, the method of driving an artificial intelligence service apparatus according to an embodiment of the present invention includes the steps of: sequentially receiving, by a communication interface unit, video frames of a captured image; and setting an event interval of an object detection error based on the confidence score obtained, and allowing the erroneous detection object of a video frame whose confidence score to exceed a threshold within the set event interval to be reused during learning.

According to an embodiment of the present invention, for example, by reusing meaningful data in which false positives or false positives occur in the post-processing process of the video detector or close thereto during learning, the accuracy of learning is increased, and the development cost due to technological improvement can be reduced.

1 is a diagram illustrating an artificial intelligence service system according to an embodiment of the present invention;
Figure 2 is a view for explaining the operation of the artificial intelligence service device of Figure 1;
3 is a block diagram illustrating the detailed structure of the artificial intelligence service apparatus of FIG. 1, and
4 is a flowchart illustrating a driving process of the artificial intelligence service device of FIG. 1 .

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

1 is a diagram illustrating an artificial intelligence service system according to an embodiment of the present invention, and FIG. 2 is a diagram for explaining the operation of the artificial intelligence service apparatus of FIG. 1 .

As shown in FIG. 1 , the artificial intelligence service system 90 according to an embodiment of the present invention includes some or all of the photographing device 100 , the communication network 110 , and the artificial intelligence service device 120 .

Here, “including some or all” means that some components such as the communication network 110 are omitted so that the photographing device 100 and the artificial intelligence service device 120 perform direct communication (eg, P2P communication), or It means that some or all of the components constituting the artificial intelligence service device 120 may be integrated into a network device (eg, a wireless switching device, etc.) constituting the communication network 110 and configured. In order to facilitate a sufficient understanding, it is explained that everything is included.

The photographing apparatus 100 may be installed in various places in an industrial site to perform photographing operations. For example, it can be installed and operated in various places for self-driving cars, smart factories, intelligent CCTV, intelligent crime analysis, and AI-based disease prediction. In addition, the photographing device 100 may be installed in a space where children are active, for example, in an arbitrary space such as a daycare center or kindergarten. Of course, the photographing apparatus 100 may be installed in a home to observe infants and nanny. The photographing apparatus 100 includes a general CCTV (Closed Circuit Television) camera or an IP (Internet Protocol) camera as a surveillance camera. In addition, the photographing apparatus 100 may include a Pan-Tilt-Zoom (PTZ) camera capable of pan, tilt, and zoom operations as well as a fixed camera. Of course, since the photographing apparatus 100 may include a camera mounted on a user terminal device such as a smartphone or a tablet PC, the embodiment of the present invention will not be particularly limited to any one form.

The photographing device 100 installed in the industrial site may be linked to the edge device. In other words, as the 4th industrial revolution era enters, the network device such as the communication network 110 or the artificial intelligence service device 120 may feel a lot of burden in arithmetic processing as the amount of data to be processed increases. To this end, by installing an edge device around the photographing device 100 to primarily perform an image analysis operation, the amount of data processing can be drastically reduced. The edge device may perform various operations in connection with the artificial intelligence service device 120, for example, by changing the setting data for analysis (eg, observation object, etc.) 120) can be provided.

The communication network 110 includes both wired and wireless communication networks. For example, a wired/wireless Internet network may be used or interlocked as the communication network 110 . Here, the wired network includes an Internet network such as a cable network or a public telephone network (PSTN), and the wireless communication network includes CDMA, WCDMA, GSM, Evolved Packet Core (EPC), Long Term Evolution (LTE), Wibro network, etc. is meant to include Of course, the communication network 110 according to the embodiment of the present invention is not limited thereto, and may be used, for example, in a cloud computing network under a cloud computing environment, a 5G network, and the like. For example, when the communication network 110 is a wired communication network, the access point in the communication network 110 can connect to a switching center of a telephone company. or by accessing various repeaters such as a Base Transmissive Station (BTS), NodeB, and e-NodeB to process data.

The communication network 110 may include an access point (AP). Here, the access point includes a small base station such as a femto or pico base station that is often installed in a building. Femto or pico base stations are classified according to the maximum number of access to the imaging device 100, etc. in the classification of small base stations. Of course, the access point may include a short-distance communication module for performing short-distance communication such as Zigbee and Wi-Fi with the photographing device 100 and the like. The access point may use TCP/IP or Real-Time Streaming Protocol (RTSP) for wireless communication. Here, short-range communication may be performed in various standards such as Bluetooth, Zigbee, infrared, radio frequency (RF) such as ultra high frequency (UHF) and very high frequency (VHF), and ultra wideband communication (UWB) in addition to Wi-Fi. Accordingly, the access point can extract the location of the data packet, designate the best communication path for the extracted location, and forward the data packet to the next device, such as the artificial intelligence service device 120 , along the designated communication path. The access point may share several lines in a general network environment, and includes, for example, a router, a repeater, and a repeater.

The artificial intelligence service device 120 may include various types of devices that execute artificial intelligence (AI) programs. For example, it may be a smartphone possessed by users, and may include a tablet PC, a desktop or laptop computer, or a smart TV or a server performing image analysis operation. Above all, the artificial intelligence service device 120 according to the embodiment of the present invention preferably includes a video detector and performs an operation for acquiring meaningful data such as false positives or false positives of an object through this. More precisely, the artificial intelligence service device 120 executes a supervised learning-based deep learning model to provide various types of services. For example, assuming that deep learning is applied to detect events and accidents, when there are false or false positives about objects within a video frame, learning accuracy can be increased by allowing learning to take place. Alternatively, the amount of training data may be increased as supervised learning data.

The artificial intelligence service device 120 according to an embodiment of the present invention extracts meaningful data, for example, data close to false positives or false positives, in a post-processing process using detectors of various devices operating in the field. Here, false positives or false positives mean a case in which an object in a video frame is erroneously detected, that is, recognized or not detected. However, in the embodiment of the present invention, it can be seen that a detection result different from that of the remaining video frames is generated for the same object within a designated time period.

Above all, in the embodiment of the present invention, the term positive event or negative event may be used. A positive event is an event determined to be true when it is assumed that binary classification is performed on an object in a video frame. If there is a frame determined to be false among the event sections determined to be true, it may be assumed to be false. On the other hand, a negative event is an event determined to be false by binary classification. If there is a frame determined to be true among the event sections determined to be false, it is assumed to be a false positive. Referring to FIG. 2 , (a) is a diagram for explaining a positive event, and (b) is a diagram for explaining a negative event. As described above, the artificial intelligence service device 120 according to the embodiment of the present invention performs image analysis on the captured image of the photographing device 100, for example, extracts objects in a video frame in the process, and tracks the extracted object. , it is possible to judge events such as incidents and accidents based on the tracking results. Typically, it will be able to judge a traffic accident or detect an act of throwing garbage without permission. However, in the embodiment of the present invention, it may be assumed as an event when a result different from other video frames is obtained in the object detection process of a designated object for a plurality of video frames.

The artificial intelligence service device 120 performs a learning operation based on supervised learning data rather than using unsupervised learning data when applying the deep learning model of artificial intelligence. Therefore, when supervised learning data is used for object analysis results of video frames, etc., it may be a false positive when it is not in the corresponding data pool or DB. Accordingly, accuracy can be improved by using the data pool for re-learning while expanding the data pool for an object (eg, a false-detection object) of a video frame in which a false positive is detected.

More specifically, as shown in FIG. 2 , the artificial intelligence service device 120 according to the embodiment of the present invention sequentially receives each video frame according to time change in order to determine false positives and false positives in a positive or negative manner. For each frame, a confidence score is calculated. Object detection in general machine learning receives a video frame (or image) and calculates a confidence score for each predefined type or class. For example, if a class such as a dog or a cat is defined and an image of a dog is input, a confidence score of 08 for dog and 0.2 for cat is calculated. This means that the probability of a dog is 0.8. In this way, a confidence score is calculated for every video frame inputted in this way, more precisely, a video frame in a section having positive or negative events. It is assumed that the positive event section is true in object detection, for example, if there is a frame determined to be false, for example, a cat, in a section where all should be determined as dogs, as false negative. Therefore, the negative event period can be regarded as the opposite concept to the positive event period. That is, if there is a frame determined to be true in a section where all should be determined to be false, it is assumed as a false positive.

Then, the artificial intelligence service device 120 extracts the event of the window by setting the size of the window, for example, the number of video frames and the threshold in the post-processing. In this process, a moving average or the like may be used. When the threshold is set to 0.75 as seen in FIG. 2 , it is determined whether the confidence score calculated from a specific video frame is less than 0.75. Alternatively, in the case of a negative, it may be determined whether it is greater than 0.75. For example, if the confidence score is lower than the threshold value in the event section estimated to be positive as in (a) of FIG. 2 , it is assumed that the frame has a high undetected probability. Conversely, if a confidence score higher than the threshold is detected in the event section that is estimated to be negative as in FIG. 2B , it is assumed that the frame has a high false positive probability.

Thereafter, the artificial intelligence service device 120 performs an operation for performing re-learning on the video frame in which false positives or false positives are detected. In other words, if data corresponding to false positives or false positives is extracted, stored, and re-learned, the performance of the detector is improved. For this purpose, in FIG. 2, for example, it is assumed that false positives or false positives occur in the 6th video frame f6 among situations of class 1 (eg, dogs), and videos of the previous and subsequent times based on the image of the corresponding video frame. An operation for re-learning is performed using the frame. It is preferable to use frames in a time interval symmetrical based on the reference frame for the previous and subsequent frames. For example, assuming that boundary clapping is set around the object, it can be used as a method of storing the median value of the boundary box of the previous video frame f4 and the subsequent video frame f8, for example, the median value of the coordinate values. In other words, the coordinates are set by matching the image of f6, and re-learning is performed based on this. For example, when there is a false detection of the same object as the corresponding false detection object through re-learning, an operation such as correcting object information or changing a state in which a boundary box is set may be performed.

Object detection in machine learning is classified as classification and localization. For example, you can find the area where an object is in the input image (eg, a boundary box (BBox)) and determine whether the object is a dog or a cat. Therefore, if it is determined that it is a cat in the image event section determined to be a dog, the corresponding boundary box is also unreliable, so the intermediate value of the boundary box before and after that, that is, the coordinate value of the boundary box can be stored and used. Using the median value of the boundary box in a video frame may be understood as forming a boundary box located at an intermediate position of the boundary box in two adjacent video frames. Through this, it is possible to improve the performance of the detector by storing the video frames in which false positives or false positives have occurred in the DB for active learning, that is, the pool and utilizing the corresponding data.

Object detection in machine learning creates a model by learning an image (or a video frame or an object within that frame), a class, and a boundary box. Therefore, by using the image of f6, the class determined at f4 and f8, and the median value of the boundary box of f4 and f8, meaningful data (eg, data close to false positives or false positives) in the post-processing process using a detector operating in the field By extracting, it will be possible to improve supervised learning-based deep learning models without high development costs as in the past.

Of course, in the embodiment of the present invention, the median value of the boundary box has been exemplified while explaining the operation of the artificial intelligence service device 120 so far, but object detection and the like can be made in various forms through pixel analysis, etc. is not particularly limited, and above all, in the embodiment of the present invention, it can be seen that a frame less than a (preset) threshold value is reused for learning in an object detection event section.

3 is a block diagram illustrating a detailed structure of the artificial intelligence service apparatus of FIG. 1 .

As shown in FIG. 3 , the artificial intelligence service device 120 of FIG. 1 according to an embodiment of the present invention includes a communication interface unit 300 , a control unit 310 , an active learning post-processing unit 320 , and a storage unit 330 . ) in part or all, and may further include a display unit.

The communication interface unit 300 receives the captured image from the photographing apparatus 100 of FIG. 1 and provides it to the control unit 310 . In this process, the communication interface unit 300 may perform operations such as modulation/demodulation, encoding/decoding, muxing/demuxing, etc., which are obvious to those skilled in the art, and thus further description will be omitted.

The control unit 310 is in charge of overall control operations of the communication interface unit 300 of FIG. 3 , the active learning post-processing unit 320 , and the storage unit 330 constituting the artificial intelligence service device 120 of FIG. 1 . Typically, the control unit 310 may temporarily store the captured image provided from the communication interface unit 300 in the storage unit 330 , then call it and provide it to the active learning post-processing unit 320 .

In addition, the control unit 310 may receive the supervised learning data to be post-processed according to the request of the active learning post-processing unit 320 and systematically classify it and store it in the DB 120a of FIG. 1 . Of course, the control unit 310 receives the analysis result of the image provided from the active learning post-processing unit 320 and stores it in the DB 120a of FIG. 1 , and in this process, the supervised learning data may be stored.

The active learning post-processing unit 320 may include a video detector and execute a supervised learning-based deep learning model. An operation of extracting meaningful data close to false positives or false positives in the post-processing process is performed using the detector, and through this, re-learning of the post-processed meaningful data is performed. Through this, the performance of the detector mounted therein can be improved.

More specifically, the active learning post-processing unit 320 calculates a confidence score for each frame when a video frame of a captured image is received. For example, if a class such as a dog or a cat is defined in advance, and an image of a dog is input, a confidence score of 0.8 is calculated, and when an image of a cat is input, a confidence score of 0.2 is calculated. This means that the probability of a dog is 0.8. The details have been sufficiently described with reference to FIG. 2 above. In post-processing, the event of the window is extracted by setting the window size, that is, the number of frames and the threshold. Here, the number of frames may refer to the number of video frames corresponding to a positive or negative period as seen in FIG. 2 . 2 shows 10 video frames.

The active learning post-processing unit 320 assumes a video frame with a high probability of missing detection when a confidence score is detected lower than the threshold in the event section estimated to be positive, and, conversely, a false positive when the confidence score is higher than the threshold value in the event section estimated to be negative. Assume that it is a video frame with high probability. Therefore, video frames less than the threshold value are reused for learning in the object detection event period. For this reuse, the active learning post-processing unit 320 matches an image for a specified class of objects with false positives or false positives, and a position value or coordinate value within a video frame for the image, for example, a supervised learning data pool. You can use it after saving it to .

In general, object detection in machine learning creates a model by learning an image, a type or class of an object, and a boundary box. Here, the boundary box may mean a position or coordinate value of an object within a video frame. Accordingly, in order to determine the correct position of the false positive or the false positive detected object, the video frame surrounding the corresponding false positive or false positive may be used by setting the coordinate value of the intermediate position using the video frame surrounding the false positive or false positive. In this way, it is possible to improve the performance of the detector by extracting, storing, and re-learning the corresponding data of false positives or false positives in the data pool.

The storage unit 330 may store various data processed under the control of the control unit 310 , for example, may temporarily store a captured image and then output it. Also, the storage unit 330 may temporarily store the supervised learning data analyzed and provided by the active learning post-processing unit 320 , and then output it to be provided to the DB 120a of FIG. 1 .

In addition to the above, detailed information regarding the communication interface unit 300, the control unit 310, the active learning post-processing unit 320, and the storage unit 330 of FIG.

On the other hand, as another embodiment of the present invention, the control unit 310 may include a CPU and a memory, and may be formed as a single chip. The CPU includes a control circuit, an arithmetic unit (ALU), an instruction interpreter and a registry, and the memory may include a RAM. The control circuit performs a control operation, the operation unit performs an operation operation of binary bit information, and the instruction interpreter performs an operation of converting a high-level language into a machine language and a machine language into a high-level language, including an interpreter or a compiler. , the registry may be involved in software data storage. According to the above configuration, for example, at the beginning of the operation of the artificial intelligence service device 120, the program stored in the active learning post-processing unit 320 is copied, loaded into a memory, that is, RAM, and then executed, thereby speeding up data operation processing. can be increased quickly.

4 is a flowchart illustrating a driving process of the artificial intelligence service apparatus of FIG. 1 .

Referring to FIG. 4 together with FIG. 1 for convenience of explanation, the artificial intelligence service device 120 according to an embodiment of the present invention sequentially receives video frames of a captured image (S400). Here, sequential reception means that, for example, if it is assumed that 60 video frames per second are received, it is received at 1/60 of a unit video frame, and it can be seen that video frames are continuously received according to time change. .

For example, referring to FIG. 2 , if f1 is a video frame received initially, it may be understood that f10 is a video frame received after 10/60 seconds. Of course, since the received video frame is determined according to the operating frequency of the artificial intelligence service device 120, it will not be specifically limited to any one form.

In addition, the artificial intelligence service device 120 sets the event interval of the object detection error based on the confidence score calculated in relation to the specified object in each received video frame, and the video in which the confidence score deviates from the threshold within the set event interval. The erroneous detection object of the frame is reused when learning (S410).

When a detection error occurs for the same object (eg, cat, or dog) extracted from each received video frame When learning an object in a video frame with a detection error based on surrounding video frames within the video frame where the detection error occurs Reuse. Of course, the detection error may indicate a frame in which a confidence score is detected to be lower than a threshold value in a positive event interval, and may indicate a video frame in which a confidence score is detected higher than a threshold value in a negative event interval.

Accordingly, it is possible to improve the performance of the detector by allowing learning to be performed even on a video frame in which false positives or false positives are detected.

In addition to the above, the artificial intelligence service device 120 of FIG. 1 may perform various operations, and since other detailed information has been sufficiently described above, it will be replaced with the contents.

On the other hand, even though it has been described that all components constituting the embodiment of the present invention are combined or operated as one, the present invention is not necessarily limited to this embodiment. That is, within the scope of the object of the present invention, all the components may operate by selectively combining one or more. In addition, although all of the components may be implemented as one independent hardware, some or all of the components are selectively combined to perform some or all functions of the combined components in one or a plurality of hardware program modules It may be implemented as a computer program having Codes and code segments constituting the computer program can be easily deduced by those skilled in the art of the present invention. Such a computer program is stored in a computer-readable non-transitory computer readable media, read and executed by the computer, thereby implementing an embodiment of the present invention.

Here, the non-transitory readable recording medium refers to a medium that stores data semi-permanently and can be read by a device, not a medium that stores data for a short moment, such as a register, cache, memory, etc. . Specifically, the above-described programs may be provided by being stored in a non-transitory readable recording medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

In the above, preferred embodiments of the present invention have been illustrated and described, but the present invention is not limited to the specific embodiments described above, and it is common in the technical field to which the present invention pertains without departing from the gist of the present invention as claimed in the claims. Various modifications may be made by those having the knowledge of, of course, and these modifications should not be individually understood from the technical spirit or perspective of the present invention.

100: photographing device 110: communication network
120: artificial intelligence service device 300: communication interface unit
310: control unit 320: active learning post-processing unit
330: storage

Claims (8)

a communication interface unit for sequentially receiving video frames of captured images; and
Based on a confidence score for each frame calculated for each video frame sequentially received according to time change with respect to a specified object in each received video frame, an object detection error that falsely detects or does not detect an object (false) ), a control unit that sets an event section indicating a temporal section, and reuses the false positive or the non-detected false detection object of a video frame in which the confidence score exceeds a threshold within the set event interval when learning; including but
The control unit is an artificial intelligence service device for determining whether there is an object detection error by comparing the sequentially received video frames with neighboring video frames to set the event period. According to claim 1,
The control unit, an artificial intelligence service device for determining the number of video frames for image analysis in order to set the event section. delete According to claim 1,
The control unit stores the image of the false detection object as supervised learning data and then reuses it during learning. 5. The method of claim 4,
The control unit further stores a coordinate value in a video frame for the erroneous detection object by matching the image, and the coordinate value is calculated based on the coordinate value of a specified object corresponding to the erroneous detection object in a surrounding video frame artificial intelligence service. 6. The method of claim 5,
The control unit extracts the coordinate values of the specified object from video frames of a previous time and a subsequent time based on the video frame of the erroneous detection object, calculates a median value of the extracted coordinate values, An artificial intelligence service device that calculates coordinate values. 6. The method of claim 5,
The control unit is an artificial intelligence service device for correcting object information for the same object of the false detection object or changing the state of a boundary box around the object according to reuse when learning the false detection object. receiving, by the communication interface unit, video frames of the captured image sequentially; and
The control unit detects an object that falsely detects or does not detect an object based on a confidence score for each frame calculated for each video frame sequentially received according to time change with respect to a designated object in each of the received video frames Set an event section indicating a temporal section of error, and reuse the false positive or the false false detection object of a video frame in which the confidence score exceeds a threshold within the set event interval when learning step; including,
determining, by the controller, whether there is an object detection error by comparing the sequentially received video frames with neighboring video frames to set the event period;
The driving method of the artificial intelligence service device further comprising.

Images