KR102182660B1 - System and method for detecting violence using spatiotemporal features and computer readable recording media storing program for executing method thereof - Google Patents

Abstract

The present invention relates to a violence detection system using a spatiotemporal feature, to a method thereof, and to a computer-readable recording medium in which a program for performing the method is stored. Specifically, the violence detection system using a spatiotemporal feature detects a person by applying a CNN model in order to reduce unnecessary frames from an acquired image, applies 3D-CNN from a frame sequence, which is a sequence of frames detected by the person to predict and determine violent activities in the image, and provides alerts to relevant security departments when predicting and determining a violent activity.

Description

A system and method for detecting violence using spatiotemporal features and computer readable recording media storing program for executing method thereof

The present invention relates to a violence detection system and method using a space-time feature, and a computer-readable recording medium in which a program for performing the method is stored. More particularly, a CNN model is applied to reduce unnecessary frames from an acquired image. Is detected, and 3D-CNN is applied from the frame sequence, which is a sequence of frames detected by humans, to extract spatio-temporal features to predict and determine violent activity in the video, and to the relevant security department when predicting and determining violent activity. It relates to a violence detection system and method using a space-time feature that provides a warning, and a computer-readable recording medium storing a program for performing the method.

In general, violence is a physical coercion that causes physical damage and exerts mental and psychological pressure. Violence occurs in various forms, such as injuring or intimidating another person, imprisoning a person, breaking into a residence, and destroying property as prescribed by law.

Violence is also increasing as society becomes group egoism, individualism, nuclear family, and psychological anxiety such as employment insecurity and academic pressure increase, and such violence occurs frequently in various places such as schools, streets, parks, medical centers, etc. have.

In order to prevent such violence and to take follow-up measures in the event of violence, numerous closed circuit televisions (CCTVs) are installed in surveillance areas such as schools, kindergartens, streets, parks, medical centers, and shelters.

Images captured through multiple CCTVs are stored on the server of the organizer who installed the CCTV, are monitored in real time by the organizer's manager, and are provided with evidence from related organizations such as the police.

However, even if the administrator monitors a large number of images captured through CCTV in real time, it is difficult for the administrator to view a large number of images at once, and it is difficult to monitor continuously for 24 hours.

In addition, even if the manager monitors the video precisely and checks the video that responds to the violence, it is difficult to determine whether this video is actually violence, and it takes a lot of time to take a lot of steps to determine and take action. There was a problem.

To solve this problem, it is based on manual functions such as Datta, et.al, Ngynen.et.al, Mahadevan.et.al, Hassner, Huang, et.al, and Guo.et.al, which detect violence from acquired images. Approaches and deep learning-based approaches such as Chen, Lloyed.et.al, Fu.et.al, and Sudhakaran.et.al have been developed.

The above-mentioned approaches have attempted to solve many problems of violence detection, including camera viewing, complex crowd patterns, and intensity variations, but when a change occurs due to the perspective, significant mutual occlusion and scale of the human body for detection of violence. There was a problem in not capturing distinctive and effective features through extraction.

In particular, conventional methods are inefficient to recognize complex patterns because they use low-level functions, and there is a problem that is difficult to implement in real-time monitoring.

In addition, conventional methods occupy more memory and take a long time due to a problem of processing a large number of non-critical frames.

In addition, conventional methods have a problem in that effective patterns cannot be learned due to lack of data and low accuracy in the violence detection benchmark data set.

Republic of Korea Patent Publication No. 10-1541272 (2015.08.03. Announcement)

Accordingly, an object of the present invention is to detect a person by applying a CNN model to reduce unnecessary frames from an acquired image, and to extract spatiotemporal features by applying a 3D-CNN from a frame sequence, which is a sequence of frames in which the person is detected. A computer-readable recording medium storing a violence detection system and method using a space-time feature that predicts and judges violent activity and provides a warning to the relevant security department when predicting and determining violent activity, and a program that performs the method. In the offering.

A violence detection system using a space-time feature according to the present invention for achieving the above object comprises: an image acquisition unit for obtaining and outputting an image photographed of a surveillance area; A person detection unit that detects a person from the image output from the image acquisition unit and captures and outputs a frame in which the person is detected; A frame sequence generator for generating and outputting a frame sequence in which the frames output from the person detection unit are grouped into a predetermined number of frames; A spatiotemporal feature extractor configured to detect a spatiotemporal feature from a frame sequence output from the frame sequence generator, classify spatiotemporal feature information according to the spatiotemporal feature information, and output information on whether violence exists in the frame sequence; And a violence detection determination unit determining whether the frame sequence is a frame sequence with violence or a frame sequence without violence based on the information on whether violence exists.

The image acquisition unit is characterized in that it acquires an image photographed through a closed circuit television (CCTV) monitoring the surveillance area.

The image acquisition unit is characterized in that the image acquisition unit acquires the image from any one of a mobile terminal capturing the surveillance area and a service application server providing an arbitrary service for the image capturing the surveillance area.

When the depth of the frame of the acquired image and the point-by-point convolution are individually calculated, the human detector includes 28 layers, and a non-linear arrangement standard and ReLU are provided after the layer excluding the last fully connected layer (FC Layer). It is characterized in that a person is detected and output for each frame by applying the applied Mobilenet??SSD CNN model.

The frame sequence generator may generate and output a frame sequence including 16 frames detected by a person.

The spatiotemporal characteristic extracting unit generates spatiotemporal characteristic information including spatial correlation information and temporal correlation information between frames detected by a person in a frame sequence input from the frame sequence generator, and a frame based on the generated spatiotemporal characteristic information It is characterized by classifying and outputting whether violence exists in the image of the sequence.

The spatiotemporal feature extraction unit includes eight convolutions (Conv), five pooling (pooling), and two fully connected layers (FC Layers) having a softmax output layer that classifies whether an image with or without violence. It characterized in that it includes.

The image acquisition unit further acquires and outputs the location information for the surveillance area, and when it is determined that the frame sequence input through the violence detection determination unit is a frame with violence, the location information of the frame sequence obtained It characterized in that it further comprises an alarm unit for notifying the violence occurrence notification information including a related organization.

The location information may be obtained from a closed circuit television management server including location information on which a closed circuit television for photographing the surveillance area is installed.

In order to achieve the above object, a method for detecting violence using spatiotemporal features according to the present invention includes: an image acquisition process in which an image acquisition unit acquires and outputs an image captured by a surveillance area; A process of extracting a person detection frame in which a person detection unit detects a person from an image output from the image acquisition unit, and captures and outputs the detected frame; A frame sequence generation process in which a frame sequence generation unit generates and outputs a frame sequence in which the frames output from the person detection unit are grouped into a predetermined number of frames; A spatio-temporal feature extraction process in which a spatiotemporal feature extractor detects a spatiotemporal feature from a frame sequence output from the frame sequence generator, classifies the spatiotemporal feature information accordingly, and outputs information on whether violence is present in the frame sequence; And a violence detection determination unit determining whether the frame sequence is a frame sequence with violence or a frame sequence without violence based on the information on whether violence exists.

The spatio-temporal feature extraction process may include: obtaining spatial correlation information between frames detected by a person in a frame sequence input from the frame sequence generator; Obtaining temporal correlation information between frames of the frame sequence; A spatiotemporal characteristic information generating step of generating spatiotemporal characteristic information including the spatial correlation information and temporal correlation information; And a classification step of classifying and outputting whether violence exists in the image of the frame sequence based on the generated spatiotemporal characteristic information.

The image acquisition process may include: an image acquisition step of obtaining an image for the surveillance area; And a location information acquisition step of acquiring location information of the surveillance area, wherein when the alarm unit determines that the frame sequence input through the violence detection determination unit is a frame with violence, the location information acquisition step of the image acquisition process It characterized in that it further comprises a warning process of notifying the violence occurrence notification information including the location information obtained by the image of the frame sequence obtained in the related organization.

A computer-readable recording medium in which a program for performing a violence detection method using a space-time feature according to the present invention to achieve the above object is: An image acquisition process in which an image acquisition unit acquires and outputs an image captured by a surveillance area. ; A process of extracting a person detection frame in which a person detection unit detects a person from an image output from the image acquisition unit, and captures and outputs the detected frame; A frame sequence generation process in which a frame sequence generation unit generates and outputs a frame sequence in which the frames output from the person detection unit are grouped into a predetermined number of frames; A spatio-temporal feature extraction process in which a spatiotemporal feature extractor detects a spatiotemporal feature from a frame sequence output from the frame sequence generator, classifies the spatiotemporal feature information accordingly, and outputs information on whether violence is present in the frame sequence; And a violence detection determination unit configured to determine whether the frame sequence is a frame sequence with violence or a frame sequence without violence based on the violence presence information. It is characterized by recording.

The present invention has the effect of reducing the total processing time by detecting a person by removing unwanted frames using an efficient CNN model in the first step.

In addition, the present invention is applied to a 3D CNN model trained on three benchmark data sets to extract spatiotemporal features, so that a pattern can be effectively learned and accuracy is improved despite a relatively small number of data sets. There is an effect that can be made.

In addition, since the present invention optimizes the model by applying the optimization module, there is an effect of increasing the speed and improving the performance in the final platform.

1 is a diagram showing the configuration of a communication system including a violence detection system using space-time features according to the present invention.
2 is a diagram showing the configuration of a violence detection system using space-time features according to the present invention.
3 is a diagram showing a violence detection process using space-time features according to the present invention.
4 is a diagram illustrating an image frame in which a person is detected in a violence detection system using space-time features according to an embodiment of the present invention.
5 is a diagram showing the structure of a 3D CNN for extracting spatiotemporal features of a violence detection system using spatiotemporal features according to the present invention.
6 is a diagram showing an image frame according to a convolution depth of a violence detection system using a space-time feature according to the present invention.
7 is a flowchart illustrating a method of detecting violence using space-time features according to the present invention.

Hereinafter, a configuration and operation of a violence detection system using space-time features according to the present invention will be described with reference to the accompanying drawings, and a violence detection method in the system will be described in detail.

1 is a diagram showing the configuration of a communication system including a violence detection system using space-time features according to the present invention.

Referring to FIG. 1, the violence detection system 300 using the space-time feature of the present invention is connected to the wired/wireless data communication network 10 and connected to the wired/wireless data communication network 10, an image providing device 100, an image providing unit 200, etc. An image may be acquired by performing data communication.

The image providing device 100 may be a plurality of closed circuit televisions (CCTVs) 110 for photographing a surveillance area at a specific location, and a mobile terminal 120 such as a smartphone or a smart pad for recording the surveillance area. There will be. The surveillance area refers to an area captured by the CCTV 110 and the mobile terminal 120.

The image providing apparatus 100 may transmit an image of the surveillance area to the violence detection system 300 of the present invention through the wired/wireless data communication network 10 or to the image providing unit 200.

The image providing unit 200 stores and manages the CCTV identification information for the CCTV 110 and the location information on which the CCTV 110 is installed, and stores images received from the CCTV 110 in the CCTV identification information and the location information. It may be a CCTV management server 210-1 that maps and stores, and provides to the violence detection system 300 of the present invention in real time, or stores and manages terminal identification information in mobile terminals 120, and photographs installed images It may include at least one application server 220 that receives an image through an application, maps the image to terminal identification information, and stores the image, and transmits the image to the violence detection system 300 of the present invention.

The mobile terminal 120 may transmit the captured image and location information of the surveillance area (location information of the mobile terminal) to one or more of the violence detection system 300 and the corresponding application server 220.

The related institution server 400 is a server installed in a related institution such as a police station, a hospital, etc., and generates an alarm through a separate alarm (not shown) when the violence detection system 300 receives information about the occurrence of violence. City) will be able to display information about the occurrence of violence. It is preferable that the violence occurrence notification information includes location information of a detection area where violence has occurred.

The violence detection system 300 acquires an image or an image including location information from any one of the image providing apparatus 100 and the image providing unit 200 described above, and analyzes the acquired image to monitor whether violence occurs, When the occurrence of violence is detected, violence occurrence notification information is transmitted to the related institution server 400 so that the generated violence can be quickly responded.

The detailed configuration and operation of the violence detection system 300 will be described in detail with reference to FIG. 2 below.

2 is a diagram showing the configuration of a violence detection system using spatiotemporal features according to the present invention, FIG. 3 is a diagram showing a violence detection process using spatiotemporal features according to the present invention, and FIG. 4 is an embodiment of the present invention. A diagram showing an image frame in which a person is detected in a violence detection system using spatiotemporal features according to the present invention, and FIG. 5 is a diagram showing the structure of a 3D CNN for extracting spatiotemporal features of a violence detection system using spatiotemporal features according to the present invention. 6 is a diagram showing an image frame according to the convolution depth of the violence detection system using the space-time feature according to the present invention. Hereinafter, it will be described with reference to FIGS. 2 to 6.

The violence detection system 300 according to the present invention includes an image acquisition unit 310, a person detection unit 320, a frame sequence generation unit 330, a spatiotemporal characteristic extraction unit 340, and a violence detection determination unit 350, and , Depending on the embodiment, it may further include an alarm unit 360 and a learning unit 380.

As shown in 511 of 510 of FIG. 3, the image acquisition unit 310 includes an image or location information from one or more of the image providing device 100 and the image providing unit 200 through the wired/wireless data communication network 10. Acquire an image and output it. The acquired image may be acquired in the form of a stream and may be output, or may be acquired in the form of image data and output, but it would be desirable to be acquired in a stream form and monitored in real time.

The person detection unit 320 monitors whether a person exists in a frame unit in the image input from the image acquisition unit 310, and outputs a corresponding frame when a person is detected.

The person detection unit 320 is a frame of an image obtained by applying a pretrained MobileNet-SSD (single shot multibox detector) convolution neural network (CNN) model as shown in 512 of FIG. Detect the person who is my object.

The MobileNet has a depth-separable convolution to detect an object instead of a regular convolution.

The MobileNet-SSD CNN model according to the present invention includes 28 layers when the frame depth and point-by-point convolution of the acquired image are individually calculated, and after the layer excluding the last fully connected layer (FC Layer). Nonlinear placement standards and ReLU apply. The ReLU is one of the activation functions widely used in machine learning.

The first convolution layer of the MobileNet-SSD CNN model according to the present invention includes two strides having a filter shape of 3*3*3*32, the input size is 224*224*3, and the next depth direction convolution There is one stride in the lusion, the filter shape is 3*3*32, and the input size is 112*112*32.

The MobileNet is mainly applied to classification, while the SSD is used to find the exact location of the multi-box detector, and the combination performs object detection. To this end, SSDs are added to the end of the network to perform feedforward convolution and create a bounding box group of a fixed size group to extract and apply a functional map to ensure the existence and detection of object instances in the box. The convolution filter and bounding box are composed of prediction classes having a probability for each class, and the class with the highest probability represents an object.

The MobileNet-SSD CNN model according to the present invention, that is, the person detection unit 320, detects people in the ice hockey image and captures and outputs the detected frames as shown in FIG. 4. In this way, since unnecessary frames are removed from the image and only frames in which only people exist, memory can be efficiently used and efficient analysis can be performed.

The frame sequence generation unit 330 collects a predetermined number or more of consecutive frames output from the person detection unit 320, and then generates and outputs the collected number of frame sequences. The frame sequence is preferably composed of 16 frames, but is not limited thereto.

The spatiotemporal characteristic extraction unit 340 receives a frame sequence output from the frame sequence generator 330, detects the spatiotemporal characteristic from the frame sequence, and classifies the spatiotemporal characteristic information accordingly, so that violence exists in the frame sequence. Prints information on whether there is violence or not.

The spatiotemporal feature extraction unit 340 extracts spatiotemporal features by applying a 3D CNN model, and outputs spatiotemporal feature information for the extracted spatiotemporal features.

The spatiotemporal feature extraction unit 340 of the present invention extracts spatial information through 2D CNN as shown in 520 of FIG. 3 and FIG. 5, and time information due to 3D convolution (Conv) and pooling (Pooling) operations. Can be extracted better. The 3D convolution of the present invention is constructed by rotating a 3D mask in a cube designed through frame assembly of a frame sequence, and a feature map obtained from the convolution layer captures motion information and is linked to a plurality of attached frames of the previous layer. . Therefore, the position, x, y, and z values in the q-th feature map of the p-th layer having the bias t _pq can be expressed by Equation 1.

는 이전 레이어에서 k 번째 특징 맵에 연결된 3D 마스크의 (a, b, c) 값이다. 커널의 가중치가 전체 큐브에 복제되므로 프레임 큐브에서 3D 컨볼루션 마스크로 한 가지 유형의 특징만 추출된다.Where Cp is the 3D mask size with time dimension,

Is the (a, b, c) value of the 3D mask connected to the k-th feature map in the previous layer. Since the weights of the kernel are replicated to the entire cube, only one type of feature is extracted from the frame cube with a 3D convolution mask.

The spatiotemporal feature extracting unit 340 receives an input frame sequence as shown in FIG. 6 and may acquire deeper features as the convolution process proceeds to the third and fifth convolution layers.

The spatiotemporal feature extraction unit 340 includes two fully connected layers (FC Layer = fc) with eight lines (convolution layer: Convolution Layer = Conv), five pooling and Softmax output layers. Includes.

Each convolutional layer has a 3*3*3 kernel with one stride, and all pooling layers are 2*2*2 kernels except the first one with a second stride with a kernel size of 1*2*2. It is Max Pooling with a size. The max pooling preserves time-based information. The number of filters for each convolution is 64, 128 and 256 for the first, second and third layers, respectively. The kernel of each convolutional layer has a temporal depth whose size is defined as D. The kernel size and padding used to apply the convolution are kept at 3 and 1 respectively. The two fully connected layers (fc6 and fc7) contain 4096 neurons, and the Softmax layer, which contains N outputs, has a different N value depending on the data set. In the present invention, since there are only two classes, violent scenes and non-violent scenes, N of the softmax layer will be 2.

The spatiotemporal feature extraction unit 340 of the present invention acquires a short frame sequence of 16 frames, but uses a random cropping size of 3*16*112*112 from the original input sequence during training, and then for effective learning, 3D convolution and pooling are performed after the frame sequence as shown in FIGS. 3 and 5.

When the result value (0 or 1) of the softmax for the frame sequence is input from the spatio-temporal characteristic extraction unit 340, the violence detection determination unit 350 indicates that the result value is violence (eg, 0 or 1) or non-violence (eg, 1). or 0), it is determined whether it is violence, and when it is determined that it is violence, the information about the occurrence of violence is output to the alarm unit 360.

When the violence occurrence information is input from the violence detection determination unit 350, the warning unit 360 generates violence occurrence notification information including the location information of the surveillance area of the frame sequence acquired through the image acquisition unit 310, and is preset. It is transmitted to the related institution server 400 to notify that violence has occurred.

The learning unit 380 learns a pattern of violent activity from a plurality of video data sets. The video data set is divided into two categories: violent class and non-violent class. That is, the learning unit 380 performs learning on both violence and non-violence. The data set could be divided into a sequence of 16 frames with an 8 frame overlay between two consecutive clips.

In addition, the learning unit 380 learns frames, frame sequences, and classified result values obtained through the image acquisition unit 310 and output through the spatiotemporal feature extraction unit 340 in real time, and optimizes the learned learning data. Thus, it may be configured to update the 3D CNN model of the spatiotemporal feature extraction unit 340.

7 is a flowchart illustrating a method of detecting violence using space-time features according to the present invention.

Referring to FIG. 7, first, the learning unit 380 of the violence detection system 300 uses as a data set a plurality of images with assault and a plurality of images prepared in advance according to the present invention. Learning by the 3D CNN model according to the invention, that is, training is performed, and learning (training) data according to the training should be registered.

The image acquisition unit 310 monitors whether an image acquisition event occurs (S111). The image acquisition event may occur at a certain period of time, may occur when an image is received from any one of the image providing device 100 and the image providing unit 200, or may be generated according to a request by the administrator. .

When an image acquisition event occurs, the image acquisition unit 310 acquires an image received from the image providing apparatus 100 and the image providing unit 200, or requests an image and receives and acquires the image accordingly, and obtains the acquired image. Output to the person detection unit 320 (S113).

When an image is input from the image acquisition unit 310, the person detection unit 320 starts monitoring the presence of a person in the image through the MobileNet-SSD CNN model according to the present invention (S115), and checks whether a person is detected ( S117).

When a person is detected, the person detection unit 117 starts to capture the frame in which the person is detected and output it to the frame sequence generation unit 330 (S119).

The frame sequence generation unit 330 determines whether a preset number of consecutive frames is input when a frame is input from the person detection unit 320 (S121), and when a preset number of frames is continuously input, the frame sequence is generated. After generation, it is output to the spatiotemporal feature extraction unit 340 (S123).

The spatiotemporal characteristic extracting unit 340 extracts the spatiotemporal characteristics of the input frame sequence, classifies whether the image of the frame sequence is an image with violence or an image without violence according to the extracted spatiotemporal characteristics, and The classification result is output to the violence detection determination unit 350 (S125).

When the classification result value is input from the spatio-temporal characteristic extraction unit 340, the violence detection determination unit 350 alerts the violence occurrence information when the result value is a result value that the image of the frame sequence is an image in which violence exists. ) Is output (S127).

When violence occurrence information is input from the violence detection determining unit 350, the warning unit 360 transmits violence occurrence notification information including the location information of the surveillance area in which the image of the frame sequence is obtained to the related institution server 400. (S129). That is, the warning unit 360 notifies that violence has occurred in a specific location (region) to a related institution such as a police station or hospital.

Meanwhile, it is common knowledge in the art that the present invention is not limited to the above-described typical preferred embodiment, but can be implemented in various ways without departing from the gist of the present invention. Anyone who has a will be able to understand it easily. If implementation by such improvement, change, substitution or addition falls within the scope of the following appended claims, the technical idea should also be considered to belong to the present invention.

10: wired/wireless data communication network 100: image providing device
200: image providing unit 300: violence detection system
310: image acquisition unit 320: human detection unit
330: frame sequence generator 340: spatiotemporal feature extraction unit
350: violence detection determination unit 360: alarm unit
380: learning department 400: related institution server

Claims (15)

An image acquisition unit for obtaining and outputting an image photographed of the surveillance area;
A person detection unit that detects a person from the image output from the image acquisition unit and captures and outputs a frame in which the person is detected;
A frame sequence generator for generating and outputting a frame sequence in which the frames output from the person detection unit are grouped into a predetermined number of frames;
A spatiotemporal feature extractor configured to detect a spatiotemporal feature from a frame sequence output from the frame sequence generator, classify spatiotemporal feature information according to the spatiotemporal feature information, and output information on whether violence exists in the frame sequence; And
And a violence detection determination unit that determines whether the frame sequence is a frame sequence with violence or a frame sequence without violence based on the information on whether violence exists.
The method of claim 1,
The image acquisition unit,
A violence detection system using a space-time feature, characterized in that acquiring an image captured through a closed circuit television (CCTV) monitoring the surveillance area.
The method of claim 1,
The image acquisition unit,
A violence detection system using a space-time feature, characterized in that the image is obtained from any one of a mobile terminal that captures the surveillance area and a service application server that provides a service application server that provides an arbitrary service for the image that the surveillance area is captured.
The method of claim 1,
The person detection unit,
When the frame depth and point-by-point convolution of the acquired image are individually calculated, 28 layers are included, and a nonlinear placement standard and ReLU are applied behind the layer excluding the last fully connected layer (FC Layer). )-Violence detection system using spatiotemporal features, characterized in that a person is detected and output for each frame by applying an SSD CNN model.
The method of claim 4,
The frame sequence generator,
A violence detection system using spatiotemporal features, characterized in that a frame sequence including 16 frames detected by a person is generated and output.
The method of claim 1 or 5,
The spatiotemporal feature extraction unit,
A person in the frame sequence input from the frame sequence generator generates spatiotemporal characteristic information including spatial correlation information and temporal correlation information between the detected frames, and violence is applied to the image of the frame sequence based on the generated spatiotemporal characteristic information. A violence detection system using spatiotemporal features, characterized in that classification and outputting whether it exists or does not exist.
The method of claim 6,
The spatiotemporal feature extraction unit,
Characterized in that it includes 8 conv, 5 pooling, and 2 fully connected layers (FC layers) with a Softmax output layer that classifies whether a video with or without violence. Violence detection system using space-time features.
The method of claim 1,
The image acquisition unit,
Further obtains and outputs location information for the surveillance area,
When it is determined that the frame sequence input through the violence detection determination unit is a frame with violence, further comprising an alarm unit for notifying a related agency of violence occurrence notification information including location information from which the image of the frame sequence is obtained. Violence detection system using space-time features characterized.
The method of claim 8,
The location information,
Violence detection system using a space-time feature, characterized in that obtained from a closed-circuit television management server including location information on which a closed-circuit television for photographing the surveillance area is installed.
An image acquisition process in which the image acquisition unit acquires and outputs an image captured of the surveillance area;
A process of extracting a person detection frame in which a person detection unit detects a person from an image output from the image acquisition unit, and captures and outputs the detected frame;
A frame sequence generation process in which a frame sequence generation unit generates and outputs a frame sequence in which the frames output from the person detection unit are grouped into a predetermined number of frames;
A spatio-temporal feature extraction process in which a spatiotemporal feature extractor detects a spatiotemporal feature from a frame sequence output from the frame sequence generator, classifies the spatiotemporal feature information accordingly, and outputs information on whether violence is present in the frame sequence; And
And a violence detection determining unit determining whether the frame sequence is a frame sequence with violence or a frame sequence without violence based on the information on whether violence is present or not.
The method of claim 10,
The spatiotemporal feature extraction process,
A spatial correlation information acquisition step of obtaining spatial correlation information between frames detected by a person in a frame sequence input from the frame sequence generator;
Obtaining temporal correlation information between frames of the frame sequence;
A spatiotemporal characteristic information generating step of generating spatiotemporal characteristic information including the spatial correlation information and temporal correlation information; And
And a classification step of classifying and outputting whether violence exists in an image of a frame sequence based on the generated spatiotemporal characteristic information.
The method of claim 10,
The image acquisition process,
An image acquisition step of obtaining an image for the surveillance area; And
Including a location information acquisition step of obtaining location information of the surveillance area,
When the warning unit determines that the frame sequence input through the violence detection determination unit is a frame with violence, a violence occurrence notification including the location information of the frame sequence obtained in the location information acquisition step of the image acquisition process Violence detection method using spatiotemporal features, further comprising an alert process of notifying information to related organizations
An image acquisition process in which the image acquisition unit acquires and outputs an image captured of the surveillance area;
A process of extracting a person detection frame in which a person detection unit detects a person from an image output from the image acquisition unit, and captures and outputs the detected frame;
A frame sequence generation process in which a frame sequence generation unit generates and outputs a frame sequence in which the frames output from the person detection unit are grouped into a predetermined number of frames;
A spatio-temporal feature extraction process in which a spatiotemporal feature extractor detects a spatiotemporal feature from a frame sequence output from the frame sequence generator, classifies the spatiotemporal feature information accordingly, and outputs information on whether violence is present in the frame sequence; And
A violence detection method using a spatiotemporal feature, characterized in that the violence detection determination unit includes a violence detection determination unit that determines whether the frame sequence is a frame sequence with violence or a frame sequence without violence based on the information on whether violence exists. A computer-readable recording medium in which a program to be executed is stored.
The method of claim 13,
The spatiotemporal feature extraction process,
A spatial correlation information acquisition step of obtaining spatial correlation information between frames detected by a person in a frame sequence input from the frame sequence generator;
Obtaining temporal correlation information between frames of the frame sequence;
A spatiotemporal characteristic information generating step of generating spatiotemporal characteristic information including the spatial correlation information and temporal correlation information; And
A computer storing a program for performing a violence detection method using spatiotemporal features, comprising a classification step of classifying and outputting whether violence exists in an image of a frame sequence based on the generated spatiotemporal characteristic information Readable recording medium.
The method of claim 13,
The image acquisition process,
An image acquisition step of obtaining an image for the surveillance area; And
Including a location information acquisition step of obtaining location information of the surveillance area,
When the warning unit determines that the frame sequence input through the violence detection determination unit is a frame with violence, a violence occurrence notification including the location information of the frame sequence obtained in the location information acquisition step of the image acquisition process A computer-readable recording medium storing a program for performing a violence detection method using a space-time feature, further comprising an alarm process of notifying information to a related organization.

Images