KR102529876B1 - A Self-Supervised Sampler for Efficient Action Recognition, and Surveillance Systems with Sampler - Google Patents

Abstract

An action recognition method of an action recognition model including a sampler according to another embodiment of the present invention comprises: a reception step of allowing the sampler trained by the above learning method to receive the down-sampled video clip; a score prediction step of allowing the trained sampler to predict irrelevance scores for each frame of all frames of the downsampled video clip; a frame removal step of removing behavior-irrelevant frames based on the predicted irrelevance score for each frame; and an action recognition step of allowing the action recognition model to perform action recognition by receiving frames selected from the frames remaining after irrelevant frames are removed from the frames of the video clip.

Description

A Self-Supervised Sampler for Efficient Action Recognition, and Surveillance Systems with Sampler}

The present invention relates to a self-map sampler for action recognition, and a video surveillance system using the sampler, and more particularly, to a self-map sampler for efficient action recognition in a real-world surveillance system, and a video surveillance system using the sampler. .

In the case of artificial intelligence systems that receive images as input and make inferences, many have been commercialized, and in certain fields, artificial intelligence that can replace experts has been developed. However, in the case of video-based artificial intelligence systems, human-level recognition performance has not yet been reached, and even if the system is built, it consumes a lot of computation compared to image-based artificial intelligence, so it is not well commercialized in real applications yet. Therefore, if a sampler capable of improving the amount of computation and recognition performance is developed, it is expected that a system based on action recognition using the sampler will have high marketability.

Behavior recognition is one of the major research topics widely applied to various robots such as video surveillance, gesture recognition, and human-robot interaction. A general prediction method of a CNN-based action recognition model is to simply receive all frames in a dense form and use the average of the predicted values output from the artificial intelligence model. However, this prediction method is inefficient because all frames are evenly utilized regardless of whether there is an action or not. Dense form prediction is more inefficient in real-time action recognition applications where the input video is unrefined. Therefore, in order to improve the prediction method of the action recognition model that receives such dense inputs, a CNN-based sampler with a very small amount of computation that selects frames related to representative actions appearing in the video is required.

Korean Patent Registration No. 10-1986002 (2019.06.04.)

S. Benaim et al., "SpeedNet: Learning the speediness in videos," in Proc.IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2020, pp. 9919??9928. D. Kim, D. Cho, and I. S. Kweon, "Self-supervised video representation learning with space-time cubic puzzles," in Proc. AAAI Conf. Artif. Intell., vol. 33, no. 01, 2019, p. 8545??8552.

Therefore, the present invention has been made to solve the problems of the prior art as described above, and an object of the present invention is to provide a self-map sampler for motion recognition and a video surveillance system using the same.

However, the object of the present invention is not limited to the above-mentioned object, and other objects not mentioned will be clearly understood by those skilled in the art from the description below.

A method for learning a CNN-based self-map sampler according to an embodiment of the present invention to solve the above problems includes receiving a first video and a second video for learning; a mixing step of extracting one irrelevant frame from the received second video and mixing the extracted frame with the first video to generate a first video having irrelevant frames; and a learning step of learning to identify which frame is a mixed frame in the mixed first video.

Preferably, the blending step randomly mixes one irrelevant frame of the received clip of the second video with the received clip of the first video to generate a clip of the first video having unrelated frames; In step 1, the sampler is trained to find the index of an irrelevant frame by taking the clip of the first mixed video as an input, the first video clip having one irrelevant frame is the input of the sampler, and the index of the frame is the target It is characterized by being a label.

를 얻는 과정 - 여기서 B는 배치(batch) 크기이고, T는 시간 차원이고, C는 채널의 수이고, H 및 W는 공간 차원으로 정의되며 -; 및 미니-배치 X 내의 각각의 비디오 클립 X_b에 대해, 시간 t에서 하나의 프레임 X_b(t)를 랜덤하게 선택하고, 그 후 그것을 시간 t에서 셔플링된 미니-배치

내의 b-번째 클립의 프레임

와 교환하는 과정;으로 얻어지는 것을 특징으로 한다.Preferably, the clips of the first video with the unrelated frames are shuffled mini-batch X along a batch axis to

The process of obtaining , where B is the batch size, T is the temporal dimension, C is the number of channels, and H and W are defined as spatial dimensions; and for each video clip X _b in mini-batch X, randomly select one frame X _b (t) at time t, then place it in a shuffled mini-batch at time t

frame of the b-th clip in

It is characterized by being obtained as; a process of exchanging with.

는 하나의 준비된 비디오 클립

를 취하고, 그 후 하나의 비디오 클립 내의 각각의 프레임에 대한 비관련성 점수

를 생성하며, 손실 함수를 기반으로 파라미터 Θ를 업데이트하는 것을 특징으로 한다.Preferably, the sampler

is one prepared video clip

, and then the irrelevance score for each frame within one video clip.

and updating the parameter Θ based on the loss function.

An action recognition method of an action recognition model including a sampler according to another embodiment of the present invention includes a receiving step of receiving a video clip in which the sampler learned by the learning method is downsampled; a score prediction step of predicting irrelevance scores for each frame of all frames of the downsampled video clip using the learned sampler; a frame removal step of removing behaviorally irrelevant frames based on the predicted irrelevance score for each frame; and a behavior recognition step of removing irrelevant frames from among the frames of the video clip and receiving selected frames among remaining frames as inputs and performing behavior recognition by the behavior recognition model.

Preferably, the frame removal step is characterized by removing frames whose predicted irrelevance score of each frame is higher than the threshold value through comparison with the threshold value.

는 5개의 3D-CNN 레이어로 구성되며,

는 비디오 클립들로 구성된 하나의 다운샘플링된 입력 미니 배치를 나타내고, 여기서 B는 배치(batch) 크기이고, T는 시간 차원이고, C는 채널의 수이고, H 및 W는 공간 차원으로 정의하면, 상기 학습된 샘플러는 하나의 미니-배치 X 내의 b번째 비디오 클립 X_b로부터의 각각의 프레임에 대한 비관련성 점수를 다음 수식과 같이 예측하며,

, 여기서

및 Θ는 각각 샘플러의 비관련성 점수 및 학습 가능한 파라미터들을 나타내고, 하나의 비디오 클립에서 각 프레임의 무관련성 점수들은 각 프레임이 비디오 클립의 전체 프레임에 얼마나 멀리 떨어져 있는지를 나타내는 것을 특징으로 한다.Preferably, the sampler network

consists of five 3D-CNN layers,

denotes one downsampled input mini-batch consisting of video clips, where B is the batch size, T is the temporal dimension, C is the number of channels, and H and W are defined as spatial dimensions: The learned sampler predicts the irrelevance score for each frame from the bth video clip X _b in one mini-batch X as follows,

, here

and Θ represent the sampler's irrelevance score and learnable parameters, respectively, and the irrelevance scores of each frame in one video clip represent how far each frame is from the entire frame of the video clip.

Preferably, among the frames of the video clip, selected frames among frames remaining after removing irrelevant frames are selected from a fixed number of N frames at regular intervals after removing frames having an irrelevance score greater than a threshold value τ. It is characterized in that the frame is selected by any one of a process or a process of selecting only frames having an irrelevance score smaller than τ after setting a threshold value τ.

Preferably, the action recognition step is characterized in that the action of the video is recognized by applying a dense prediction method using the selected frame as an input.

A video surveillance system using a sampler according to another embodiment of the present invention includes an object detection unit that receives video from CCTV and detects people in the video using an object detection model used for real-time object detection to limit the location; an object tracking unit recognizing an individual ID of each detected person using an object tracking model used for object tracking; a self-map sampler that receives a video clip consisting of a plurality of frames for each recognized ID, removes shuffled frames from the video clip, and outputs the remaining frames as a motion recognition model; and an action recognition model for recognizing each person's action using the remaining frames received from the self-map sampler.

Advantageously, the sampler receives a downsampled video clip, predicts irrelevance scores for each frame of full frames of the downsampled video clip, and determines whether an action is determined based on the predicted irrelevance score for each frame. A process of removing relevant frames is performed, and the action recognition model removes irrelevant frames from among frames of the video clip and receives selected frames among the remaining frames as input and performs action recognition by the action recognition model.

Preferably, the sampler is characterized in that it is a sampler learned by the learning method.

In the method of operating a video surveillance system using a sampler according to another embodiment of the present invention, an object detection unit receives a video from a CCTV, detects people in the video using an object detection model used for real-time object detection, and locates the location. object detection step to limit; an object tracking step in which an object tracking unit recognizes an individual ID of each detected person using an object tracking model used for object tracking; a frame selection step in which the self-map sampler receives a video clip composed of a plurality of frames for each recognized ID, removes shuffled frames from the video clip, and outputs the remaining frames as a motion recognition model; and an action recognition step in which the action recognition model recognizes each person's action using the remaining frames received from the self-map sampler.

Advantageously, the sampler receives a downsampled video clip, predicts irrelevance scores for each frame of full frames of the downsampled video clip, and determines whether an action is determined based on the predicted irrelevance score for each frame. A process of removing relevant frames is performed, and the action recognition model removes irrelevant frames from among frames of the video clip and receives selected frames among the remaining frames as input and performs action recognition by the action recognition model.

A computer-readable recording medium according to another embodiment of the present invention is characterized in that a program for executing the learning method of the CNN-based self-map sampler or the behavior recognition method of the behavior recognition model including the sampler is recorded.

When an experiment was conducted on an abnormal behavior detection dataset in a subway station published by AI Hub using the self-map sampler for behavior recognition according to an embodiment of the present invention, prediction was made through only about 1/3 of the amount of computation in the same behavior recognition model. It was able to perform, and the performance was also improved by about 7.5%.

When the self-guided sampler for action recognition of the present invention is used, the amount of computation of an artificial intelligence model based on video input to be researched and developed in the future can be greatly reduced, reducing the burden on real-time performance and computational efficiency when building a system. In addition, the recognition performance of the action recognition model can be expected to be improved.

1 shows a process of generating training data for the sampler of the present invention.
Figure 2 shows the entire pipeline of action recognition using the sampler of the present invention during the inference step.
3 illustrates a learning method of a self-map sampler according to another embodiment of the present invention.
4 illustrates a behavior recognition method of a behavior recognition model including a sampler according to another embodiment of the present invention.
5 is a block diagram of a real-time monitoring system in a subway station using a magnetic map sampler according to another embodiment of the present invention.
6 illustrates a method of operating a real-time monitoring system in a subway station using a magnetic map sampler according to another embodiment of the present invention.

Since the present invention can apply various transformations and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all conversions, equivalents, or substitutes included in the spirit and scope of the present invention. In describing the present invention, if it is determined that a detailed description of related known technologies may obscure the gist of the present invention, the detailed description will be omitted.

Terms used in this application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, the term "comprises" or "consists of" is intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other It should be understood that the presence or addition of features, numbers, steps, operations, components, parts, or combinations thereof is not precluded.

The present invention relates to a CNN-based sampler with a very small amount of computation that selects a frame related to a representative action appearing in a video in order to improve a prediction method of an action recognition model receiving dense inputs.

The CNN-based sampler of the present invention is learned by mixing an unrelated frame from an input video, that is, a frame extracted from another video, and finding it. Using the sampler learned in this way, frames to be used as input to the action recognition model are refined. Sampler can be combined with any action recognition model and greatly improves the efficiency of prediction time and memory.

The CNN-based sampler of the present invention is applied to an artificial intelligence network that receives a CNN-based action recognition model and a video as an input. For most artificial intelligence models, a high-performance graphic processing unit is essential, and the size of the artificial intelligence model that can be loaded into the system varies depending on the performance of the graphic processing unit. In the case of artificial intelligence that receives video as an input, it has a very large amount of computation compared to image-based artificial intelligence as it captures motion information on the time axis and recognizes actions based on this. Therefore, when developing and researching real applications based on behavioral recognition artificial intelligence, the amount of computation is one of the key factors to consider. In the case of the CNN-based sampler model of the present invention, this problem can be greatly alleviated, and the performance of the action recognition model can be improved.

Hereinafter, the motivation for the invention of the self-guided sampler for action recognition of the present invention, network structure, sampler training procedure, and action recognition using the sampler will be described.

The motivation of the present invention is that there are overlapping or irrelevant frames among the input frames for the action recognition model. Therefore, removing redundant or irrelevant frames from an input video clip can reduce the amount of computation and improve accuracy. To achieve this goal, the sampler model of the present invention is trained to measure the relevance of each frame to the entire clip. In addition, since the sampler is an auxiliary model for faster inference, the sampler is designed to be lightweight. In order to train such a sampler model with supervised learning, expensive annotations to irrelevant frame locations within the videos are required. Instead of these expensive annotations, the present invention uses a simple self-teaching method to synthetically create a video with unrelated frames by blending: given two videos, one frame from the first video is replaced by the second video. , and a sampler model is trained to identify which frames are blended frames in the second video. At this time, human annotation is not required. After training, our sampler model is expected to be able to find irrelevant frames in real video.

Describe the network structure.

는 5개의 3D-CNN 레이어로 구성된다.

는 비디오 클립들로 구성된 하나의 다운샘플링된 입력 미니 배치를 나타낸다. B는 배치(batch) 크기이고, T는 시간 차원이고, C는 채널의 수이고, H 및 W는 공간 차원이다. 샘플러는 하나의 미니-배치 X 내의 b번째 비디오 클립 X_b로부터의 각각의 프레임에 대한 비관련성 점수를 다음 수학식 1과 같이 생성한다: Sampler network of the present invention

is composed of five 3D-CNN layers.

represents one downsampled input mini-batch composed of video clips. B is the batch size, T is the temporal dimension, C is the number of channels, and H and W are the spatial dimension. The sampler generates an irrelevance score for each frame from the bth video clip X _b in one mini-batch X as:

및 Θ는 각각 샘플러의 비관련성 점수 및 학습 가능한 파라미터들을 나타낸다. 하나의 비디오 클립에서 각 프레임의 무관련성 점수들은 각 프레임이 비디오 클립의 전체 프레임에 얼마나 멀리 떨어져 있는지를 나타낸다. 이러한 무관련성 점수들은 행동 인식을 위해 프레임을 제거하는 데 사용된다.here

and Θ represent the sampler's irrelevance score and learnable parameters, respectively. The irrelevance scores of each frame in a video clip represent how far each frame is from the overall frames of the video clip. These irrelevance scores are used to remove frames for action recognition.

The sampler training procedure is described.

We describe a self-supervised learning method to create a sampler model that can find irrelevant frames without human effort. To do this, one unrelated frame is randomly mixed into a given video clip to create the sampler's input clip. Then train the sampler to find the indices of irrelevant frames. That is, a video clip with one irrelevant frame is the input, and the index of that frame is the target label.

1 shows a process of generating training data for the sampler of the present invention. As shown in Figure 1, we mix a random number of frames at random locations in the same mini-batch.

를 얻는다. 미니-배치 X 내의 각각의 비디오 클립 X_b에 대해, 시간 t에서 하나의 프레임 X_b(t)를 랜덤하게 선택하고, 그 후 그것을 도 1에 도시된 바와 같이 시간 t에서 셔플링된 미니-배치

내의 b-번째 클립의 프레임

와 교환한다. 이 프로세스는 다음 수학식 2와 같이 표현됩니다.Specifically, by shuffling X along the batch axis,

get For each video clip X _b in mini-batch X, randomly select one frame X _b (t) at time t, and then divide it into a shuffled mini-batch at time t as shown in FIG.

frame of the b-th clip in

exchange with This process is expressed as Equation 2 below.

및

는 샘플러를 훈련하기 위한 입력들 및 라벨들이다. Y_b는 각각의 비디오 클립

에 대해 원-핫 벡터(one-hot vector)이고, Y_b(t)는 프레임을 교환할 때 1이며 그 외는 0이 된다. 원-핫 인코딩은 타겟 레벨 집합의 크기를 벡터의 차원으로 하고, 라벨이 해당하는 인덱스의 값에 1을 부여하고 다른 인덱스에는 0을 부여하는 벡터 표현 방식이다. 이렇게 표현된 벡터를 원-핫 벡터(One-Hot vector)라고 한다.here,

and

are the inputs and labels for training the sampler. Y _b is for each video clip

is a one-hot vector for , and Y _b (t) is 1 when exchanging frames and 0 otherwise. One-hot encoding is a vector representation method in which the size of a target level set is the dimension of a vector, and 1 is assigned to an index corresponding to a label and 0 is assigned to other indices. A vector expressed in this way is called a one-hot vector.

는 하나의 준비된 비디오 클립

를 취하고, 그 후 하나의 비디오 클립 내의 각각의 프레임에 대한 비관련성 점수

를 생성한다. 다음 수학식 3의 손실 함수를 기반으로 파라미터 Θ를 업데이트한다.In the training stage, the sampler of the present invention

is one prepared video clip

, and then the irrelevance score for each frame within one video clip.

generate The parameter Θ is updated based on the loss function of Equation 3 below.

Figure 2 shows the entire pipeline of action recognition using the sampler of the present invention during the inference step. As shown in Figure 2, the sampler of the present invention receives as input a downsampled video clip and then produces irrelevance scores. We remove behaviorally irrelevant frames based on the predicted irrelevance score for each frame. Finally, action recognition is performed using only the remaining selected frames.

Referring to FIG. 2, action recognition using a sampler will be described in more detail.

에 입력하고, 훈련된 샘플러는 각각의 프레임에 대한 무관련성 점수들을 획득한다. 추론 단계에서의 샘플러의 입력이 훈련 단계의 입력과 다르지만, 훈련된 샘플러는 여전히 비디오 클립에서 각 프레임을 전체 프레임에 대한 무관련성에 대해 점수화할 수 있다. 마지막으로, 입력 비디오 클립에서 비관련성 점수가 높은 프레임을 제거한 다음, 나머지 프레임을 액션 인식 모델의 입력으로 제공한다. 특히 두 가지 버전의 프레임 제거 방법을 사용한다. 첫 번째 방법(V1)은 임계값 τ보다 큰 비관련성 점수를 갖는 프레임을 제거한 후 일정 간격(오름차순으로 비관련성 점수를 정렬한 후)으로 N개의 고정된 수의 프레임을 선택하는 것이다. 두 번째 방법(V2)은 임계값 τ를 설정한 후, 비관련성 점수가 τ보다 작은 프레임들만을 사용하는 것이다. 선택된 프레임을 사용하여 기존의 조밀 예측 방법(dense prediction method)을 적용하여 비디오의 행동을 인식한다. 조밀 예측 방법은 행동 인식에서 통상적으로 사용되는 예측 방법으로서, 전체 비디오를 어떤 길이의 클립들로 분할하고, 클립들 모두를 행동 인식 모델에 제공하고 모든 예측을 평균하는 기법이다.The ultimate goal of the sampler of the present invention is to improve the performance of motion recognition models in terms of both accuracy and computational cost. To this end, all frames from one video clip are sampled using the trained sampler of the present invention.

, and the trained sampler obtains irrelevance scores for each frame. Although the sampler's input in the inference phase is different from the input in the training phase, the trained sampler can still score each frame in the video clip for irrelevance to the entire frame. Finally, frames with high irrelevance scores are removed from the input video clip, and the remaining frames are provided as input to the action recognition model. In particular, it uses two versions of the frame removal method. The first method (V1) is to select a fixed number of N frames at regular intervals (after sorting the irrelevance scores in ascending order) after removing frames with irrelevance scores greater than the threshold τ. The second method (V2) sets a threshold value τ, and then uses only frames with an irrelevance score smaller than τ. Using the selected frame, the existing dense prediction method is applied to recognize the behavior of the video. The dense prediction method is a prediction method commonly used in action recognition, and is a technique of dividing an entire video into clips of a certain length, providing all of the clips to an action recognition model, and averaging all predictions.

3 illustrates a learning method of a self-map sampler according to another embodiment of the present invention.

As shown in FIG. 3, the CNN-based self-mapped sampler learning method includes a receiving step (S100), a mixing step (S200), and a learning step (S300).

In the receiving step (S100), the first video and the second video are received for learning on the self-learning sample.

In the mixing step (S200), one unrelated frame is extracted from the received second video, and the extracted frame is mixed with the first video to generate a first video having unrelated frames. In the mixing step, one unrelated frame of the received clip of the second video is randomly mixed with the received clip of the first video to generate a clip of the first video having unrelated frames.

In the learning step (S300), it is learned to identify which frame is a mixed frame in the mixed first video. In the learning step, the sampler is trained using the mixed clip of the first video as an input to find an index of an irrelevant frame.

를 얻는 과정 - 여기서 B는 배치(batch) 크기이고, T는 시간 차원이고, C는 채널의 수이고, H 및 W는 공간 차원으로 정의되며 -; 및 미니-배치 X 내의 각각의 비디오 클립 X_b에 대해, 시간 t에서 하나의 프레임 X_b(t)를 랜덤하게 선택하고, 그 후 그것을 시간 t에서 셔플링된 미니-배치

내의 b-번째 클립의 프레임

와 교환하는 과정;으로 취득하게 된다. 상기 샘플러

는 하나의 준비된 비디오 클립

를 취하고, 그 후 하나의 비디오 클립 내의 각각의 프레임에 대한 비관련성 점수

를 생성하며, 손실 함수를 기반으로 파라미터 Θ를 업데이트 한다.The first video clip with the one irrelevant frame is the input of the sampler, and the index of that frame becomes the target label. A clip of the first video with the unrelated frames is obtained by shuffling the mini-batch X along the batch axis.

The process of obtaining , where B is the batch size, T is the temporal dimension, C is the number of channels, and H and W are defined as spatial dimensions; and for each video clip X _b in mini-batch X, randomly select one frame X _b (t) at time t, then place it in a shuffled mini-batch at time t

frame of the b-th clip in

The process of exchanging with; the sampler

is one prepared video clip

, and then the irrelevance score for each frame within one video clip.

and update the parameter Θ based on the loss function.

4 illustrates a behavior recognition method of a behavior recognition model including a sampler according to another embodiment of the present invention.

As shown in FIG. 4, the behavior recognition method of the behavior recognition model including the sampler includes a receiving step (S1100), a score prediction step (S1200), a frame removal step (S1300), and an action recognition step (S1400). .

In the receiving step (S1100), the sampler learned by the learning method of FIG. 3 receives the downsampled video clip.

는 5개의 3D-CNN 레이어로 구성되는데,

는 비디오 클립들로 구성된 하나의 다운샘플링된 입력 미니 배치를 나타내고, 여기서 B는 배치(batch) 크기이고, T는 시간 차원이고, C는 채널의 수이고, H 및 W는 공간 차원으로 정의하면, 상기 학습된 샘플러는 하나의 미니-배치 X 내의 b번째 비디오 클립 X_b로부터의 각각의 프레임에 대한 비관련성 점수를 수학식 1과 같이 예측하고, 하나의 비디오 클립에서 각 프레임의 무관련성 점수들은 각 프레임이 비디오 클립의 전체 프레임에 얼마나 멀리 떨어져 있는지를 나타낸다.In the score prediction step (S1200), the learned sampler predicts irrelevance scores for each frame of all frames of the downsampled video clip. the sampler network

is composed of five 3D-CNN layers,

denotes one downsampled input mini-batch consisting of video clips, where B is the batch size, T is the temporal dimension, C is the number of channels, and H and W are defined as spatial dimensions: The learned sampler predicts the irrelevance score for each frame from the bth video clip X _b in one mini-batch X as shown in Equation 1, and the irrelevance scores of each frame in one video clip are respectively Indicates how far apart the frames are in the entire frame of the video clip.

In the frame removal step (S1300), action irrelevant frames are removed based on the predicted irrelevance score for each frame. In the frame removal step, frames having a predicted irrelevance score higher than the threshold value are removed through comparison with the threshold value.

In the action recognition step (S1400), the action recognition model performs action recognition by receiving selected frames among the remaining frames after removing irrelevant frames from among the frames of the video clip. A process of selecting a fixed number of N frames at regular intervals after removing frames having an irrelevance score greater than a threshold τ, or a threshold After setting τ, it is a frame selected by any one of the processes of selecting only frames with a non-correlation score smaller than τ. In the action recognition step, a video action is recognized by applying a dense prediction method using the selected frame as an input.

Hereinafter, a monitoring system using the magnetic map sampler of the present invention will be described.

5 is a block diagram of a real-time monitoring system in a train station using the magnetic map sampler of the present invention. As shown in FIG. 5, the real-time monitoring system at a subway station using the magnetic map sampler of the present invention includes a magnetic map sampler 100, an action recognition model 200, an object detector 300, and an object tracking unit 400. It is composed by

As shown in FIG. 5, the monitoring system according to another embodiment of the present invention is intended to quickly respond to various accidents that frequently occur in subways with minimal human labor. This monitoring system recognizes various behaviors such as normal, stair fall, escalator fall, fainting, vandalism, assault, and theft, and warns the user in case of an abnormal situation. Unlike existing benchmark datasets and behavior recognition methods for behavior recognition that recognize video-level behavior, the surveillance system of the present invention recognizes individuals' behavior in a moving picture (video).

Therefore, in the present surveillance system, the object detection unit 300 and the object tracking unit 400 are used to track individuals in the video. To build such a system, we collect datasets for action recognition, object detection, and object tracking tasks, train models for tasks, and integrate the trained models into a subway station monitoring system.

The reason for using samplers in real-world scenarios in the present invention will be explained. The surveillance system of the present invention is expected to be able to receive video streams from CCTV 500 and recognize individual behavior in real time. However, in untrimmed (refined) video, redundant overlapping frames may occupy a majority of the video. Therefore, in a real scenario, a sampler to trim the video is needed. The sampler can be used to filter out failed tracking frames and redundant frames, making the final prediction more accurate and more computationally efficient.

The pipeline of the real-time monitoring system of the present invention is shown in FIG. First, the object detection unit 300 detects and localizes people using an object detection model such as the YOLOv5s model used for real-time object detection, and the object tracking unit 400 uses a DeepSort model used for object tracking Recognizes the individual ID of each detected person using an object tracking model such as Then, each person's action is recognized by using 16 frames for each extracted ID as input to the motion recognition model. However, since the performance of the object detection unit 300 and the object tracking unit 400 are not perfect, the IDs of 16 frames are not exactly the same, and if the mixed IDs are not removed, the prediction is the red line (Rline) in FIG. will be inaccurate as Therefore, as shown in the green line (Gline) of FIG. 5, shuffled frames are removed through the self-map sampler, and calculation and performance-efficient prediction are performed.

To build a deep learning-based surveillance system, a sufficient amount of high-quality training dataset is essential. There are several datasets built in subway stations, but there are still a lack of datasets covering abnormal behavior recognition and vulnerable object tracking in subway stations due to privacy, security, access rights to CCTV, accidents, etc. Therefore, high-risk accidents that frequently occur in subway stations are investigated, and abnormal behaviors and vulnerable targets (objects) are defined. After that, more than 60 CCTVs in the subway station are used to build an abnormal behavior recognition dataset and a vulnerable object tracking dataset.

The deviant behavior recognition dataset consists of approximately 7,300 five-minute clips of 13 deviant behaviors. In the CCTV 500 of FIG. 3, a video frame having a resolution of 3,840×2,160 is obtained. There is also a bounding box containing over 1 million objects for unusual behavior. Abnormal behaviors defined in the data set include hidden camera sexual misconduct, escalator falls, environmental-induced falls, stair falls, drunkenness, loitering, fainting, vandalism, kidnapping, assault, theft, misdirection of approach, and unauthorized entry.

The vulnerable object tracking dataset is composed of a process similar to the abnormal behavior recognition dataset. The vulnerable object tracking dataset consists of approximately 7,000 clips of approximately 5 minutes in length for six types of objects: wheelchair, blind, wanderer, stroller, infant, and child. There are over 250,000 labeled objects.

The abnormal behavior recognition dataset and the vulnerable object tracking dataset are divided into a training set and a test set at a ratio of 7:3 based on the number of clips.

Referring to FIG. 5, implementation details of the real-time monitoring system of the present invention will be described.

The surveillance system of the present invention is designed to obtain 3,840×2,160 resolution input video from one CCTV and process it using one Nvidia RTX 3090 GPU. For real-time object detection, the YOLOv5s model is used. The model is trained with a combination of CrowdHuman, WidePerson and the object tracking dataset as a training dataset. The final model used for the surveillance system achieves an AP of 92.0 in the test partition of the object tracking dataset. DeepSort is adopted as an object tracking model, and the Market-1501 dataset and the object tracking training dataset are used as training datasets. When the YOLOv5s model and the DeepSort model are used together, the performance of 72 MOTA is achieved on the object tracking test dataset, and the object tracking test dataset is used in the surveillance system.

In order to apply the self-guided sampler of the present invention to a monitoring system, the sampler is trained using the proposed abnormal behavior recognition training dataset. Specifically, downsampling the input size to 128x128, passing it to the sampler, and utilizing random cropping and rotation as augmentation during training.

In the present invention, an experiment for a self-supervised sampler V1 is designed by inputting 64 consecutive video frames to the sampler and extracting 16 fixed frames based on the result. In the magnetic map sampler V2 experiment, a maximum of 64 and a minimum of 16 frames from video frames are input.

Finally, the MARS model is used for real-time action recognition without using optical flow. For MARS training, we use kinetics400 pre-trained weights and only fine-tune the last fully connected layer using the anomalous action recognition training dataset. The trained MARS model achieves 94% performance when using the ID and the bounding box GT (ground truth) of the person.

To verify the effectiveness of self-mapped samplers in a surveillance system in subway stations, dense, uniform, random, self-mapped samplers-V1 and V2 algorithms were compared in an abnormal behavior recognition test set. Table 1 shows the performance of all algorithms (methods) compared for each deviant behavior classification (class). When using the GT box and ID of each object, write "with GT".

As shown in Table 1 (quantitative results (all evaluations were performed using 3D-Resnext101 in the abnormal behavior recognition dataset), when the object detection unit and the object tracking unit are used without using GT, the ID is Since it is mixed, it can be seen that the performance of dense, uniform, and random sampling deteriorates, especially the normal classification shows 100% performance when using GT tracking.

If GT is not used, performance drops by more than 30% for dense, uniform, and random sampling strategies. Normal classification means that the performance of the object detection unit and the object tracking unit is more sensitive than other classifications. Surprisingly, when the self-guided samplers are used (both V1 and V2), the performance for normal classification again reaches 100%. This indicates that the sampler, which is also the object of the present invention, effectively removes IDs mixed in clips. Considering that it is important to minimize the false alarm rate since most behavior cases in real-world surveillance images are normal, the self-map sampler of the present invention can be considered sufficiently useful.

6 illustrates a method of operating a real-time monitoring system in a subway station using a magnetic map sampler according to another embodiment of the present invention.

As shown in FIG. 6, the method of operating the real-time monitoring system in a subway station using a self-map sampler includes an object detection step (S2100), an object tracking step (S2200), a frame selection step (S2300), and an action recognition step (S2400). consists of including

In the object detection step (S2100), the object detection unit receives a video from a CCTV, detects people in the video using an object detection model used for real-time object detection, and limits the location.

In the object tracking step (S2200), the object tracking unit recognizes an individual ID of each detected person using an object tracking model used for object tracking.

In the frame selection step (S2300), the self-map sampler receives a video clip composed of a plurality of frames for each recognized ID, removes shuffled frames from the video clip, and outputs the remaining frames as a motion recognition model. The sampler receives a downsampled video clip, predicts irrelevance scores for each frame of the entire frame of the downsampled video clip, and removes behaviorally irrelevant frames based on the predicted irrelevance score for each frame. carry out the process

In the action recognition step (S2400), the action recognition model recognizes each person's action using the remaining frames received from the self-map sampler. The action recognition model removes irrelevant frames from among the frames of the video clip and receives selected frames among the remaining frames as inputs, and the action recognition model performs action recognition.

As described above, the present invention has been described by the limited embodiments and drawings, but the present invention is not limited to the above embodiments, and those skilled in the art in the field to which the present invention belongs can make various modifications and transformation is possible Therefore, the spirit of the present invention should be grasped only by the claims described below, and all equivalent or equivalent modifications thereof will fall within the scope of the spirit of the present invention.

In addition, the sampler learning method and behavior recognition method according to the present invention can be implemented as computer readable codes on a computer readable recording medium. A computer-readable recording medium includes all types of recording devices that store data that can be read by a computer system. Examples of the recording medium include ROM, RAM, CD ROM, magnetic tape, floppy disk, optical data storage device, hard disk, flash drive, etc., and also implemented in the form of a carrier wave (for example, transmission through the Internet). include In addition, the computer-readable recording medium is distributed to computer systems connected through a network, and computer-readable codes can be stored and executed in a distributed manner.

Claims (14)

In the learning method of a CNN-based self-mapped sampler,
A receiving step of receiving a first video and a second video for learning;
a mixing step of extracting one irrelevant frame from the received second video and mixing the extracted frame with the first video to generate a first video having irrelevant frames; and
It is configured to include; a learning step of learning to identify which frame is a mixed frame in the mixed first video,
The mixing step
randomly mixing one unrelated frame of the received clip of the second video with the received clip of the first video to generate a clip of the first video having unrelated frames;
The clip of the first video with the unrelated frames is
By shuffling the mini-batch X along the batch axis,

The process of obtaining , where B is the batch size, T is the temporal dimension, C is the number of channels, and H and W are defined as spatial dimensions; and
For each video clip X _b in mini-batch X, randomly select one frame X _b (t) at time t, and then place it in a shuffled mini-batch at time t

frame of the b-th clip in

It is obtained by the process of exchanging with;
The exchange process is expressed as the following formula,

here,

and

are the inputs and labels for training the self-supervised sampler, and Y _b is each video clip

A learning method for a CNN-based self-mapped sampler, characterized in that it is a one-hot vector for and Y _b (t) is 1 when frames are exchanged and 0 otherwise.
According to claim 1,
The learning stage is
The self-mapped sampler is trained using the clip of the first mixed video as input to find the index of an unrelated frame, the first video clip having one irrelevant frame is the input of the self-mapped sampler, and the index of the frame Learning method of a CNN-based self-mapped sampler, characterized in that is a target label.
delete According to claim 2,
The magnetic map sampler

is one prepared video clip

, and then the irrelevance score for each frame within one video clip.

A method for learning a CNN-based self-mapped sampler, characterized in that for generating and updating the parameter Θ based on the loss function.
In the action recognition method of an action recognition model including a self-guided sampler, the self-guided sampler learned by the learning method having the characteristics of claim 1,
a receiving step of receiving the downsampled video clip;
a score prediction step of predicting irrelevance scores for each frame of all frames of the downsampled video clip;
a frame removal step of removing action-irrelevant frames based on the predicted irrelevance score for each frame; and
An action recognition step of removing irrelevant frames from among the frames of the video clip and receiving selected frames among the remaining frames as input and performing action recognition by the action recognition model; Behavior recognition method.
The method of claim 5, wherein the frame removal step
An action recognition method of an action recognition model including a self-map sampler, characterized in that by comparing with a threshold value, frames with predicted irrelevance scores of each frame are higher than the threshold value are removed.
According to claim 6,
Magnetic Map Sampler Network

consists of five 3D-CNN layers,

denotes one downsampled input mini-batch consisting of video clips, where B is the batch size, T is the temporal dimension, C is the number of channels, and H and W are defined as spatial dimensions: The learned self-map sampler predicts the irrelevance score for each frame from the bth video clip X _b in one mini-batch X as follows,

here

and Θ represent the irrelevance score and learnable parameters of the self-map sampler, respectively, and the irrelevance scores of each frame in one video clip represent how far each frame is from the entire frame of the video clip. Action recognition method of action recognition model including map sampler.
According to claim 5,
Among the frames of the video clip, the selected frame among the frames remaining after removing irrelevant frames is
A process of selecting a fixed number of N frames at regular intervals after removing frames with an irrelevance score greater than the threshold τ, or selecting only frames with an irrelevance score less than τ after setting a threshold τ An action recognition method of an action recognition model including a self-map sampler, characterized in that the frame selected by any one of the processes.
The method of claim 5, wherein the action recognition step
An action recognition method of an action recognition model including a self-map sampler, characterized in that the action of the video is recognized by applying a dense prediction method using the selected frame as an input.
In a video surveillance system using a magnetic map sampler,
An object detection unit that receives a video from a CCTV and detects people in the video using an object detection model used for real-time object detection to limit a location;
an object tracking unit recognizing an individual ID of each detected person using an object tracking model used for object tracking;
It is learned by the learning method having the characteristics of claim 1, receives a video clip composed of a plurality of frames for each recognized ID, removes shuffled frames from the video clip, and outputs the remaining frames as a motion recognition model. sampler; and
A video surveillance system using a self-map sampler, characterized in that it comprises a; action recognition model for recognizing each person's action using the remaining frames received from the self-map sampler.
According to claim 10,
The self-map sampler receives a downsampled video clip, predicts irrelevance scores for each frame of the entire frame of the downsampled video clip, and behaviorally irrelevant frames based on the predicted irrelevance score for each frame. perform the process of removing
The video surveillance system using a self-mapped sampler, characterized in that the action recognition model performs action recognition by receiving selected frames among the remaining frames after removing irrelevant frames from among the frames of the video clip.
delete In the operating method of a video surveillance system using a magnetic map sampler,
An object detection step in which an object detection unit receives a video from a CCTV and detects people in the video using an object detection model used for real-time object detection to limit a location;
an object tracking step in which an object tracking unit recognizes an individual ID of each detected person using an object tracking model used for object tracking;
The self-guided sampler learned by the learning method having the characteristics of claim 1 receives a video clip consisting of a plurality of frames for each recognized ID, removes shuffled frames from the video clip, and outputs the remaining frames as a motion recognition model. a frame selection step; and
A method of operating a video surveillance system using a self-map sampler, comprising: an action recognition step in which the action recognition model recognizes each person's action using the remaining frames received from the self-map sampler.
According to claim 13,
The self-map sampler receives a downsampled video clip, predicts irrelevance scores for each frame of the entire frame of the downsampled video clip, and behaviorally irrelevant frames based on the predicted irrelevance score for each frame. perform the process of removing
The method of operating a video surveillance system using a self-map sampler, characterized in that the action recognition model performs action recognition by receiving selected frames among the remaining frames after removing irrelevant frames from among the frames of the video clip.

Images