KR102090739B1 - Intellegent moving monitoring system and the method thereof using video region grid multidivision for video image similarity-analysis - Google Patents

Abstract

The present invention relates to an intelligent movement monitoring system using a video area grid multi-segmentation for similarity analysis of a video. The intelligent movement monitoring system using a video area grid multi-segmentation for similarity analysis of a video comprises: a video photographing part photographing a video of an object to be monitored and includes a plurality of frames photographed by a mobile camera; a feature point extraction part extracting feature points of the video of the object to be monitored; a video index DB in which the feature points of the video are indexed and stored; and a video analysis part comparing similarities by comparing the feature points of the video of the object to be monitored and feature points corresponding to the video of the same path previously stored in the video index DB.

Description

INTELLEGENT MOVING MONITORING SYSTEM AND THE METHOD THEREOF USING VIDEO REGION GRID MULTIDIVISION FOR VIDEO IMAGE SIMILARITY-ANALYSIS}

The present invention relates to an intelligent mobile monitoring system using similarity analysis to compare screens of digital image files and a monitoring method thereof. In particular, the present invention relates to a system capable of automatically dividing a digital photographed image file into grids and using it to output an event such as an image and a warning at a time point when an abnormality is found, and a monitoring method thereof.

Recently, the role of surveillance cameras is increasing day by day due to the increase in crimes and safety accidents, and furthermore, it has become a very effective means for arresting criminals after death.

Accordingly, the number of surveillance cameras installed is rapidly increasing, and the economic burden is also increasing. This was due to the fact that the surveillance camera is constantly photographing a fixed area, or even if the angle can be changed, the surveillance camera is fixed and the shooting area is limited.

In addition, it was ultimately the responsibility of the person to find any abnormal signs through the surveillance camera, which was inconvenient for the observer to be constantly monitoring the images captured by the surveillance camera.

In order to solve this inconvenience, in Korean Patent Registration No. 10-1734029, "Intelligent Mobile Surveillance System Using Similarity Analysis of Mobile-Sliding Video and Its Monitoring Method," sub-video finger print for each frame image of a video that captures an area to be monitored Create a video finger print on a video file by grouping the sub video finger prints, and compare the feature points extracted with the sub video prints with the indexed feature points of the image stored in the image index DB, and shoot at the surveillance object. It is judging whether the area in the video is abnormal.

However, the patent registration compares the feature points extracted by the sub-video finger print generated for each image frame with the feature points for each image frame in the previous database DB, so even if a large difference occurs in some areas of the image frame There is a problem in that it is not possible to detect that an abnormal situation has occurred because it is determined that there is no significant difference in the whole because the comparison is performed for each image frame.

Registered Patent No. 10-1734029-0000 (2017.05.01)

The present invention is to divide an image frame into sub-regions, extract feature points from the divided regions, and compare the divided regions so that it can be easily detected when a change in some region of the monitored region occurs.

The present invention is a video recording unit for photographing a video that is a surveillance target image including a plurality of frames photographed with a mobile camera; A feature point extracting unit for extracting a feature point of the image of the surveillance object; A video index DB in which feature points of a video are indexed and stored; An image analysis unit that analyzes similarity by comparing the feature points of the image photographed with the surveillance object and the feature points corresponding to the video of the same path pre-stored in the image index DB. The present invention relates to an intelligent mobile monitoring system using segmentation.

In addition, in the present invention, the feature point extracting unit decodes the video stream of the monitored image into an RGB format (including an infrared image format) or a black and white image format, and then divides each decoded frame image into a plurality of sub-regions. After dividing each sub-area into a plurality of block units, calculating the average value of the image brightness data of each pixel for each of the divided block units, extracting a feature point based on a difference between the average values of the image brightness data between adjacent blocks Fingerprints grouped by sub-regions of a frame may be generated.

In addition, in the present invention, the image analysis unit compares the feature points extracted with the fingerprint grouped for each sub-region image of each frame from the image captured by the surveillance object and the indexed feature points pre-stored in the image index DB, in the same place. It is possible to search a shooting time point of, and determine whether the similarity of feature points corresponding to sub-region images of a plurality of frames in a predetermined section is greater than or equal to a threshold value from the searched shooting time point.

In addition, in the present invention, the image analysis unit, a part of the feature point values extracted by the fingerprint grouped for each sub-region image of each frame from the image capturing the surveillance object, and a part of the indexed feature point values previously stored in the image index DB When the comparison values are in a certain range by comparing each other, the feature points of the sub-region images for each frame are compared by synchronizing the captured images and the images previously stored in the image index DB, and the difference between the feature points is greater than or equal to a threshold value. When a sub-region exists, it is possible to detect that another event has occurred in the captured image.

In addition, in the present invention, the manager inspecting unit for displaying the image and the similarity of the point in time at which the similarity point of the feature point is less than a threshold value and receiving the presence or absence of specificity of the image is further included. It may further include; an image index unit for storing the identified feature points of the monitored image in the image index DB.

In addition, the present invention comprises the steps of preparing an image index DB containing the feature points and peculiarities for each frame of the image determined as having no peculiarities; An image photographing unit photographing an image to be monitored; Transmitting, by the feature point extraction unit, feature points and elapsed time values for each sub-region image of each frame extracted from the image captured by the monitoring object to the image analysis unit; Searching the image index DB for feature points for each sub-region image of each frame transmitted by the image analysis unit; Determining whether the degree of similarity is greater than or equal to a threshold value based on feature points corresponding to sub-region images of a plurality of frames in a predetermined section; As a result of the determination, the image analysis unit performs multi-segmentation of the image area grid to analyze the similarity of the image including the step of determining if there is a singularity when the similarity is less than the threshold, and setting the comparison target image if there is no singularity. It relates to an intelligent mobile monitoring method to use.

In the present invention, compared to extracting the feature points for each frame, the frames are divided into sub-regions, and the respective divided regions are compared to facilitate detection when a change in a portion of the monitored region occurs.

1A is a block diagram of the present invention.
1B is another block diagram of the present invention.
FIG. 2 is a conceptual diagram of an intelligent mobile surveillance system using the similarity analysis of a mobile photographed image of the present invention to detect specific features at a specific photographing time point through a feature point of a monitored image.
3 is an example of dividing the frame 100 of the mobile photographed image of the present invention into sub-frames.
4 is an example of dividing into a plurality of blocks in order to extract singularities from sub-frames [SI (1,1)].
5 is an example of extracting singularities in each of the sub-frames of the frame 100.
6A and 6B are comparative examples of singularities of the present invention.
5A to 5D are comparative examples of singularities according to actual images in the conventional invention.
6A to 6B are comparative examples of singularity in the present invention.
7A to 7D are comparative examples of singularity in the conventional invention.
8A to 8B are comparative examples of singularity in the present invention.
9A to 9C are comparative examples of singularity in the conventional invention.
10A to 10B are examples of singularity in the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. First, it should be noted that in adding reference numerals to the components of each drawing, the same components have the same reference numerals as possible, even if they are displayed on different drawings. In describing the present invention, when it is determined that detailed descriptions of related known configurations or functions may obscure the subject matter of the present invention, detailed descriptions thereof will be omitted.

Figure 1a is a block diagram showing an intelligent mobile surveillance system using the similarity analysis of the moving image of the present invention, the image capture unit 200, feature point extraction unit 210, image analysis unit 220, image index DB ( 260).

The image photographing unit 200 may be a camera or the like capable of photographing various videos, and transmits an image to be monitored to the feature point extraction unit 210.

Meanwhile, the feature point extraction unit 210 extracts a video fingerprint, which is a video feature point, by analyzing video data included in a video file.

Here, the video finger print refers to a kind of video unique ID element capable of uniquely identifying video data, and the feature point extraction unit 210 selectively separates only video data from the image file before extracting the video finger print. You can precede the process.

The feature point extractor 210 selectively separates a video stream from a video file to extract a video finger print, and decodes the separated video stream into an RGB format. Subsequently, each frame 100 constituting the decoded image data is divided into a plurality of sub-regions, and sub-video feature points are extracted for each of the divided sub-region images.

The method of extracting the sub-video feature points in the sub-region of each frame 100 is, for example, narrow paper and one other paper (Job Oostveen, Ton Kalker, Jaap Haitsma's paper "Feature Extraction and a Database Strategy for Video" Fingerprinting "(Lecture Notes in Computer Science 2314 Springer 2002).

First, the feature point extraction unit 210 is a plurality of sub-regions (SI _(1,1) , SI _(1,2) , SI _{(1,3) ...} ] And each of the divided sub-regions can be divided into a plurality of block units (FIG. 4 is divided into 9 blocks) as shown in FIG. 4.

For example, each sub-region [SI _(1,1) , S _(1,2) , S _{(1,3) ...} ] _divides the vertical axis of the image into n and divides the horizontal axis into m equals the sub-region image of the frame. Is divided into n × m blocks.

Then, the feature point extracting unit 210 calculates the average of the Y values, which are image brightness data for each divided block unit, and then generates a sub-video fingerprint of a predetermined number of bits based on the difference in the average image brightness values between adjacent blocks.

[Equation 1]

An example of a bit allocation function 'B (r, c, p)' that can be used to generate a sub video fingerprint for a sub-region image of one frame.

Here, B (r, c, p) is a function for allocating bits for blocks located in the r-th and c-th positions, respectively, based on the horizontal and vertical axes of the p-th sub-region image [for example, SI _{(1, p)} ]. . In addition, F (r, c, p) represents the average image brightness values of blocks located at the r-th and c-th, respectively, based on the horizontal and vertical axes of the p-th sub-region image.

In F (r, c, p) of [Equation 1], r has an integer value from 1 to n, and c has an integer value from 1 to m-1. Here, a is a number smaller than 1 and is a constant that can be appropriately selected.

Meanwhile, the feature point extracting unit 110 may generate a sub video fingerprint by using Cb or Cr values, which are color difference data of each pixel, in addition to Y values, which are brightness data of an image.

The feature point extracting unit 210 extracts a sub video fingerprint for a sub-region image of each frame constituting a video, and then groups each extracted sub video fingerprint to a video file including each frame and each frame. Complete the creation of a video fingerprint for.

In the present invention, any algorithm can be applied as long as it can extract video fingerprints other than those described in the narrow paper above and one paper.

On the other hand, when the similarity of the feature point information is greater than or equal to a threshold value as a result of determining whether the similarity of the feature point information is less than a threshold value, the image capturing unit 200 further includes a control unit to control the position of the image capturing unit 200 to be fixed. . The control unit performs a function of controlling a flight position of a drone or the like, and by fixing the drone or the like in place as soon as it is discovered, it is possible to keep an eye on the occurrence of the anomaly.

 The feature point extraction unit 210 may use a color histogram graph to extract color-based feature points such as fire. This can use the content of Wei-Lun Chao's article "The Core of Video Fingerprinting Example of Feature Extraction".

Also, the feature point extraction unit 210 may use an object recognition-based feature point extraction algorithm to extract feature points based on object movement, such as intrusion detection. It is preferable to use Huitao Luo's paper (Algorithms for Video Ob ject Detection and Segmentation with Application to Content-Based Multimedia Systems).

Meanwhile, the feature point extraction unit 210 transmits the eigenvalues of each frame extracted from the sub-video feature points to the image analysis unit 220. The image analysis unit 220 may search from a specific time point according to the transferred parameter or transfer factor.

The image analysis unit retrieves the feature points extracted from the existing images from the image index DB 260 to obtain mutual similarity between the received video feature points through a bit operation, and analyzes a frame including a sub-region image that is below a threshold value preset by an administrator. (220). The video feature point (= finger print) can be numerically or encrypted for each sub-region as shown in FIG. 5.

As illustrated in FIG. 5, finger prints are generated as P _{(1, 1)} , P _{(1, 2)} , P ₍₁ , ₃₎ , and P _{(1, 4)} ... for each sub region, For example, P _{(1, 1)} can be set by numerically quantifying the RGB values of a plurality of block units _shown in FIG. 4.

6A to 6B are examples of comparing an image captured by the present invention with an image stored in the image index DB.

The five frames (100ba, 100bb, 100bc, 100bd, and 100be) in FIG. 6A are real-time video recordings, and are video sets of frames taken during a specific period T ₁ hour. Assuming that the starting point of the photographing is the Ti time, each of the five frames is finger-printed.

In addition, the 8 frames in FIG. 6B are pre-stored images, and 5 frames (5) each having a fingerprint value and a threshold value or more of a sub-region image of the 5 frames (100ba, 100bb, 100bc, 100bd, 100be) 100aa, 100ab, 100ac, 100ad, 100ae) and a specific section T _S.

That is, the finger print values P ₍₁ , ₁₎ , P _{(1, 2)} , P ₍₁ , ₃₎ , and P _{(1, 4} ) of each sub-region of five frames (100ba, 100bb, 100bc, 100bd, 100be) _{) ...} Fingerprint values P _(1,1) , P _{(1, 2)} , P _(1,3) of 5 frames (100aa, 100ab, 100ac, 100ad, 100ae) of DB where some of them are already stored , If it is found to correspond to a part of P _{(1, 4) ..} above a threshold value, the starting points of the similarity comparison are synchronized as Ti and Ts, and the feature points of the sub-region image included in each subsequent frame are mutually compared. Will be.

In order to synchronize, not all of the fingerprint values of the sub-regions of the frames are matched, but when the number of fingerprints of a certain number of frames and the number of frames of each sub-frames are synchronized, synchronization is performed.

For example, the finger print values of the sub-regions SI _(1,1) and SI _(2,1) of the first to fifth frames (100ba, 100bb, 100bc, 100bd, and 100be) of the captured image are stored. Frames (100aa, 100ab, 100ac, 100ad, 100ae) of sub-regions SI _(1,1) and SI _(2,1) difference in finger print value difference is less than 10% (threshold is 90%) shooting 5 frames (100ba, 100bb, 100bc, 100bd, 100be) and 5 pre-stored frames (100aa, 100ab, 100ac, 100ad, 100ae) can be synchronized.

After the synchronization, the finger print values of the sub-regions of each frame are compared, and it can be seen that an event different from normal occurs when an error of 10% or more occurs in some parts.

For example, in the process of comparing the fourth frame 100bd and the fifth frame 100be of the captured image and the fourth frame 100ad and the fifth frame 100ae of the pre-stored image, the sub-region SI _{(6,1 )} , When the difference in the values of finger print values P _(6,1) , P _(2,2) and P _{(2, 4)} in SI _{(2, 2)} and SI _(2,4) is greater than 10% , It can be seen that other events have occurred in the sub-regions SI _(6,1) of the fourth frame 100bd and SI _{(2, 2)} and SI _(2,4) of the fifth frame 100be.

That is, it is possible to automatically recognize that an abnormality has occurred in the background corresponding to the above 4th and 5th frames when the unmanned aerial vehicle photographs the changing background.

By quantifying the feature points corresponding to each sub-region of one frame and comparing a plurality of frames with each other, it is to find out peculiarities in each sub-region of a plurality of frames rather than a fixed one image.

Preferably, the image index DB 260 is preferably indexed according to the elapsed time, frame, and GPS coordinates. In particular, in the case of using GPS coordinates, if the camera position has been moved or if the GPS coordinates exist even if the route is modified, it can be used when the routes overlap using the coordinates.

In the image analysis unit 220, some images whose similarity is detected as a threshold value or more are determined as images without specifics that photograph the same place of the monitored image requesting analysis, and some images below the threshold value capture the same place. It is determined as an image having one specific feature, and outputs a part or a captured image of the monitored image requested for analysis through the index value of the frame obtained from the image analysis unit 220 to the manager inspection unit 230 of FIG. 1B.

Alternatively, even if the threshold value is greater than or equal to the threshold value, it is preferable to determine that the corresponding image is unusual when it is retrieved from the image index DB 260 with the unusual characteristic, and to transmit and output it to the administrator inspection unit together with the specific information recorded by the administrator separately.

Meanwhile, it is preferable that the image analysis unit 220 performs a function of searching for the image index DB 260 by limiting to a predetermined period. It is possible to increase the search speed by limiting to the search section preset by the administrator without searching for the entire data previously stored in the image index DB 260.

FIG. 2 is a conceptual diagram of an intelligent mobile surveillance system using the similarity analysis of a mobile photographed image of the present invention to detect specific features at a specific photographing time point through a feature point of a monitored image.

As shown in FIG. 2, a surveillance target image photographed on September 10 or currently being photographed in real time through a drone or the like is compared with an image photographed on September 9. The comparison target is not an image of the image itself, but the computation speed is fast because the feature-printed feature points extracted by the feature point extraction unit 210 are compared with each other through a bit operation. Therefore, real-time comparison is possible. The comparison is performed by the image analysis unit 220.

As shown in FIG. 2, as a result of mutual comparison of feature points from A point to D point of the image, the similarity is more than a threshold value, and it will be determined that there is no specificity. However, at the E point, a fire will occur and the similarity will be less than the critical value, and it will be determined as the occurrence of an unusual feature.

7A to 7D are comparisons of images for each frame, and when an image having a large fire as shown in FIG. 7A is compared with a previous image of the image index DB of FIG. 7B, an abnormality is detected by measuring similarity of 79%, When a small fire occurs in some corners of the frame image as shown in FIG. 7C, when compared with the previous image of the image index DB of FIG. 7D, it is measured with similarity of 95% and does not detect an abnormal situation.

As described above, when comparing the images by extracting the singularity for each previous frame, there is a problem in that it is impossible to accurately detect a change in an object to be photographed due to errors such as a photographing position and a camera angle.

In contrast, in the present invention, since the singularity, which is fingerprinted by dividing by sub-region in each frame, is extracted and stored, the singularities are compared for each sub-region, it is possible to more accurately detect a change in an object.

For example, when a fire occurs only in the sub-region SI _(4,5) as in FIG. 8B, when compared to the sub-region SI _(4,5) of the corresponding image index DB in FIG. ) To detect changes.

However, when comparing them frame by frame, the difference is only in the sub-regions SI _(4,5) in the entire frame, and it is recognized as a similarity of 95% or more as a whole, and a change cannot be detected.

In addition, FIGS. 9B and 9C show 80% similarity due to a position error when a photographed image is shifted compared to a video frame (FIG. 9A) of the image index DB, such as SSI01 and SSI02, to detect that a change has occurred.

However, as shown in FIGS. 10A and 10B, when the corresponding sub-regions are found and compared, the sub-regions SI _(1,11) of FIG. 10A and the sub-regions SI ( _{1, 10} ) of FIG. 10B are compared, and in this case, the similarity is 95%. You will be informed that there is no abnormality.

As described above, the singularity is extracted for each sub-region of the captured image frame and compared with the sub-region of the stored image in the image index DB, thereby increasing the accuracy of the change.

100: video frame 200: video recording unit
210: feature point extraction unit 220: image analysis unit
260: image index DB

Claims (8)

An image photographing unit for photographing a video which is a surveillance target image including a plurality of frames photographed by a mobile camera;
A feature point extracting unit for extracting a feature point of the image of the surveillance object;
A video index DB in which feature points of a video are indexed and stored;
Includes a; image analysis unit that analyzes the similarity by comparing the feature points corresponding to the video of the same path pre-stored in the image index DB and the feature points of the image captured by the monitoring object;
The feature point extraction unit,
The monitoring of the video stream of the object image the RGB format, the infrared image format or after decoding the black-and-white image format of each frame image of the decoded multiple sub-regions _{[SI (1,1), SI (} 1,2), SI (1 _{, 3) ...} ], and then divide each of the plurality of sub-regions into a plurality of block units, calculate the average value of the image brightness data of each pixel for each of the divided block units, and then image brightness between adjacent blocks Characteristic points are extracted based on the difference in the average values of data to generate fingerprints [P _(1,1) , P _{(1, 2)} , and P _{(1,3) ..} ] grouped by sub-regions of each frame,
The image analysis unit
_Feature points extracted with fingerprints [P _(1,1) , P _{(1, 2)} , and P _{(1,3) ..} ] grouped by sub-region images of each frame from the captured image of the surveillance object and the image An intelligent mobile surveillance system that uses multi-segmentation of the image area grid to analyze the similarity of images, which compares the indexed feature points stored in the index DB with each other to search for the shooting point in the same place. According to claim 1,
The image analysis unit,
Intelligent movement using multi-segmentation of the image area grid to analyze the similarity of the image, characterized in that it determines whether the similarity of the feature points corresponding to the sub-region images of each of a plurality of frames in a predetermined section is greater than or equal to a threshold value from the searched shooting time point Surveillance system. According to claim 1,
The image analysis unit,
The comparison value is within a certain range by comparing a portion of the feature point values extracted with the fingerprint grouped by the sub-region image of each frame from the image of the surveillance object and a portion of the indexed feature point value previously stored in the image index DB. If present, compare the feature points of each sub-frame image for each frame by synchronizing the pre-stored image in the image index DB and the captured image.
An intelligent mobile monitoring system using multi-segmentation of the image area grid to analyze the similarity of an image, characterized in that when a difference between the feature points and a sub-region where a difference value is greater than or equal to a threshold is detected, another event has occurred in the captured image. According to claim 2,
It further includes a manager inspecting unit that displays an image and a similarity at the time of photographing when the similarity of the feature points is less than a threshold value, and is assigned whether or not there are any particulars of the image;
The manager inspection unit,
An intelligent movement monitoring system using multi-segmentation of the image area grid for similarity analysis of images, further comprising: an image indexing unit that stores the feature points of the image to be monitored that are not unusual in the image index DB. . Preparing an image index DB containing feature points and singularities for each frame of the image determined as having no peculiarities;
An image photographing unit photographing an image to be monitored;
Transmitting, by the feature point extraction unit, feature points and elapsed time values for each sub-region image of each frame extracted from the image captured by the monitoring object to the image analysis unit;
Searching the image index DB for feature points for each sub-region image of each frame transmitted by the image analysis unit;
Determining whether the degree of similarity is greater than or equal to a threshold value based on feature points corresponding to sub-region images of a plurality of frames in a predetermined section;
As a result of the determination, the image analysis unit determines that there is an unusual feature when the similarity is less than a threshold value,
If there are no unusual items, it includes a setting step of setting the video to be compared.
The feature point extraction unit,
The monitoring of the video stream of the object image the RGB format, the infrared image format or after decoding the black-and-white image format of each frame image of the decoded multiple sub-regions _{[SI (1,1), SI (} 1,2), SI (1 _{, 3) ...} ], and then divide each of the plurality of sub-regions into a plurality of block units, calculate the average value of the image brightness data of each pixel for each of the divided block units, and then image brightness between adjacent blocks Characteristic points are extracted based on the difference in the average value of data to generate fingerprints [P _(1,1) , P _{(1, 2)} , P _{(1,3) ..} ] grouped by sub-regions of each frame,
The image analysis unit
_Feature points extracted with fingerprints [P _(1,1) , P _{(1, 2)} , and P _{(1,3) ..} ] grouped by sub-region images of each frame from the captured image of the surveillance object and the image An intelligent movement monitoring method using multi-segmentation of an image area grid for similarity analysis of images, by comparing the indexed feature points previously stored in the index DB and searching for a shooting time point in the same place. The method of claim 5
The image analysis unit,
Intelligent movement using multi-segmentation of the image area grid to analyze the similarity of the image, characterized in that it determines whether the similarity of the feature points corresponding to the sub-region images of each of a plurality of frames in a predetermined section is greater than or equal to a threshold value from the searched shooting time point Monitoring method. The method of claim 5
The image analysis unit determines whether an image having a similarity and specificity above a threshold can be searched in a predetermined section from the entire image index DB, which is a past photographed image, and a determination result, if the search is possible, corresponds to the predetermined section An intelligent movement monitoring method using multi-segmentation of an image area grid for analyzing the similarity of an image, further comprising outputting an image, similarity, and peculiar contents to the inspector. The method of claim 5,
The image analysis unit,
The comparison value is within a certain range by comparing a portion of the feature point values extracted with the fingerprint grouped by the sub-region image of each frame from the image of the surveillance object and a portion of the indexed feature point value previously stored in the image index DB. If present, compare the feature points of each sub-frame image for each frame by synchronizing the pre-stored image in the image index DB and the captured image.
An intelligent movement monitoring method using multi-segmentation of the image area grid for similarity analysis of an image, characterized in that when a difference between the feature points and a sub-region where a difference value is greater than or equal to a threshold exists, another event has occurred in the captured image.

Images