KR20230022010A - Method of controlling sports activity classification learning apparatus, computer readable medium and apparatus for performing the method - Google Patents

Abstract

Provided is a sports activity classification learning device, which includes: a database module which matches sports activity image information with sports activity classification information and stores matching data as learning data; a frame extraction module which extracts a frame; an important frame extraction module which extracts important frames from the extracted frames; a spatial information extraction module which extracts spatial information of important frames; a feature vector acquisition module which acquires feature vectors of important frames; and a sports activity classification module which classifies into sports activities.

Description

Control method of sports activity classification learning device, recording medium and device for performing the same

The present invention relates to a control method of a sports activity classification learning device, a recording medium and an apparatus for performing the same, and more particularly, to a sports activity classification learning device for classifying sports activities by inputting at least one frame from sports activity image information. It relates to a control method, a recording medium and a device for performing this.

Activity recognition technology is a technology that recognizes what kind of activity is occurring and classifies the recognized activity.

Activity recognition may be performed in a sensor-based manner. Sensor-based activity recognition uses sensors such as an accelerometer and a gyro sensor. However, in sporting events, athletes are prohibited from wearing any electronic equipment, including sensors such as accelerometers and gyroscopes. Therefore, there is a limitation that it is difficult to use the sensor-based activity recognition method in the field of sports activity recognition.

As a result, various deep learning techniques such as Deep Neural Network (DNN), Convolutional Neural Network (CNN), Deep Belief Network (DBN), and Recurrent Neural Network (RNN) have been developed. It is required to recognize sports activities of players by applying them to image information of a sports game, and to classify the recognized sports activities.

A technical problem to be solved by the present invention is a control method of a sports activity classification learning device that extracts a frame from video information of a sports game, obtains a temporal attention score, acquires a spatial attention map, and classifies sports activities. It is to provide a recording medium and a device to perform.

One aspect of the present invention is a sports activity classification learning device for extracting at least one frame from sports activity image information and classifying the sports activity, wherein the sports activity image information and the sports activity classification information are matched and stored as learning data. database module; a frame extraction module extracting at least one frame from the image information stored in the database module; an important frame extraction module extracting important frames from the extracted frames; a spatial information extraction module extracting spatial information of the important frame; a feature vector acquisition module for receiving the spatial information of the important frame as an input value of a Long Short Term Memory (LSTM) model and acquiring a feature vector of the important frame; and a sports activity classification module that obtains a probability value from the feature vector of the important frame and classifies it into the sports activity matched with the image information constituting the important frame only when the probability value is greater than or equal to a preset probability value. can

The important frame extraction module may include a pixel extraction unit which extracts R channel pixels, G channel pixels, and B channel pixels from the frame; a pixel summing unit summing the extracted pixels; A pixel pooling operation unit that performs MAX-POOLING and AVERAGE-POOLING operations on the sum of pixels; Each of the MAX-POOLING operation value and the average pooling operation value a weight setting unit configured to set a weight, wherein the sum of the weight set for the maximum pooling (MAX-POOLING) and the weight set for the average pooling (AVERAGE-POOLING) is set to 1; and a temporal attentional attention score acquisition unit that obtains a temporal attentional attention score using a value obtained by summing the maximum pooling operation value and the average pooling operation value.

Also, the important frame extraction module may include extracting, as important frames, a frame having a relatively large value multiplied by the temporal attentional attention score obtained from the temporal attentional attention score obtaining unit.

The temporal attention score obtaining unit may include: a first fully connected layer to which a value obtained by summing the MAX-POOLING calculation value and the AVERAGE-POOLING calculation value is input; a ReLU activation function performer receiving an output value output from the first Fully Connected Layer as an input value; a second Fully Connected Layer to which an output value output from the ReLU activation function performer is input as an input value; and a sigmoid activation function execution unit to which an output value output from the second Fully Connected Layer is input as an input value.

In addition, the spatial information extraction module may include: a first convolution layer inputting the important frame and outputting a feature map; a feature map pooling operation unit that performs a MAX-POOLING operation and an AVERAGE-POOLING operation on the feature map; a channel attention map obtaining unit acquiring a channel attention map from an output value of the feature map pooling operation unit; and a spatial attention map obtaining unit acquiring a spatial attention map from the channel attention map.

Another aspect of the present invention is a control method of a sports activity classification learning device for extracting at least one frame from sports activity image information and classifying the sports activity, wherein the sports activity image information and the sports activity classification information are matched for learning. data, at least one frame is extracted from the sports activity image information stored as the learning data, an important frame is extracted from the extracted frame, spatial information of the important frame is extracted, and spatial information of the important frame is extracted. is input as an input value of a Long Short Term Memory (LSTM) model, a feature vector of the important frame is obtained, a probability value is obtained from the feature vector of the frame, and only when the probability value is greater than or equal to a preset probability value It may include classifying the sports activity matched with the image information constituting the important frame.

In addition, extracting an important frame from the extracted frame extracts an R channel pixel from the frame, extracts a G channel pixel, extracts a B channel pixel, sums the extracted pixels, and The maximum pooling (MAX-POOLING) and average pooling (AVERAGE-POOLING) operations are performed on the sum of , The sum of the weight set for the maximum pooling (MAX-POOLING) and the weight set for the average pooling (AVERAGE-POOLING) is set to 1, and the maximum pooling (MAX-POOLING) calculation value and the average pooling (AVERAGE-POOLING) This may include obtaining a temporal attention score by using a value obtained by summing POOLING) calculation values.

In addition, obtaining a temporal attention score using the sum of the MAX-POOLING operation value and the average pooling operation value is the average of the MAX-POOLING operation value and the average pooling operation value. The sum of the AVERAGE-POOLING operation values is input to the first fully connected layer as an input value, and the output value output from the first fully connected layer is input to the ReLU activation function performing unit as an input value, and the ReLU An output value output from the activation function execution unit may be input as an input value to the second fully connected layer, and an output value output from the second fully connected layer may be input as an input value to the sigmoid activation function execution unit.

In addition, in extracting the spatial information of the important frame, the frame multiplied by the temporal attention score is input to the first convolution layer to output a feature map, and a MAX-POOLING operation and averaging are performed on the feature map. AVERAGE-POOLING operation is performed, and a channel attention map is obtained using the value obtained by performing the MAX-POOLING operation and the AVERAGE-POOLING operation of the feature map. and obtaining a spatial attention map from the channel attention map.

In another aspect of the present invention, a computer program for performing a control method of a sports activity classification learning device may be recorded in a computer-readable storage medium.

According to one aspect of the present invention described above, by providing a control method for a sports activity classification learning device for classifying sports activities, a recording medium and an apparatus for performing the same, sports activities are performed in consideration of both the temporal attention score and the spatial attention map. activities can be categorized.

1 is a conceptual diagram illustrating a sports activity classification learning device according to an embodiment of the present invention.
2 is a conceptual diagram illustrating an important frame extraction module.
3 is a conceptual diagram showing how a plurality of frames are multiplied by temporal attention scores through a TAM calculation process.
4 is a conceptual diagram illustrating a process of obtaining a spatial attention map through a 2D CNN model.
5 is a conceptual diagram illustrating a process of classifying sports activities after passing through a 2D CNN model.
6 is a diagram showing a table in which the sports activity classification learning apparatus of the present invention is evaluated through mAP.
7 is a flowchart illustrating a control method of a sports activity classification learning device according to an embodiment of the present invention.
8 to 9 are flowcharts illustrating a process of acquiring temporal attention points.
10 is a flowchart illustrating a process of obtaining a spatial attention map.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The detailed description of the present invention which follows refers to the accompanying drawings which illustrate, by way of illustration, specific embodiments in which the present invention may be practiced. These embodiments are described in sufficient detail to enable one skilled in the art to practice the present invention. It should be understood that the various embodiments of the present invention are different from each other but are not necessarily mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in another embodiment without departing from the spirit and scope of the invention in connection with one embodiment. Additionally, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the invention. Accordingly, the detailed description set forth below is not to be taken in a limiting sense, and the scope of the present invention, if properly described, is limited only by the appended claims, along with all equivalents as claimed by those claims. Like reference numbers in the drawings indicate the same or similar function throughout the various aspects.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the drawings.

1 is a conceptual diagram illustrating a sports activity classification learning device according to an embodiment of the present invention.

The sports activity classification learning device 1 includes a database module 10, a frame extraction module 20, an important frame extraction module 30, a spatial information extraction module 40, a feature vector acquisition module 50, and a sports activity classification module 60.

The sports activity classification learning device 1 may be a device that extracts at least one frame from sports activity image information and learns to classify which sports activity is occurring from the extracted frames. At this time, the sports activity may be an activity in which a batter swings in a baseball game, an activity in which a batter hits a home run or a strike, and the like, an activity in which a player scores a goal in a soccer game, an activity in which a shooter takes a corner kick or a free kick. This could include activities being played, activities being fouled, activities being fouled, activities being taken penalty kicks, etc. However, examples of sports activities have been given, but are not limited thereto.

The database module 10 may match sports activity image information with sports activity classification information and store them as learning data. For example, if the sports activity image information is image information of a batter swinging in a baseball game, the sports activity classification information can be matched with 'swing' and stored, and the sports activity image information is an image of a batter hitting a home run in a baseball game. If it is information, sports activity classification information may be matched with 'home run' and stored.

The frame extraction module 20 may extract one or more frames from image information stored in the database module 10 . For example, at least one continuous frame or at least one discontinuous frame may be extracted at preset time intervals from image information of a batter hitting a home run in a baseball game.

The critical frame extraction module 30 may calculate a temporal attention score from the pixel values of the frame and multiply the frame by the temporal attention score. The method of calculating the temporal attention score may calculate the temporal attention score through a Temporal Attention Module (TAM). At this time, TAM (Temporal Attention Module) is a method for extracting important frames and non-important frames by calculating importance between a plurality of frames.

For example, if the value of the second frame multiplied by the temporal attention score is greater than the value of the first frame multiplied by the temporal attention score, the second frame is extracted as an important frame and the first frame is regarded as a non-important frame. can be extracted.

A detailed description of the Temporal Attention Module (TAM) will be described later in detail with reference to FIGS. 2 and 3 .

The spatial information extraction module 40 may obtain spatial attention maps of important frames and non-important frames multiplied by temporal attention scores.

In the present invention, a spatial attention map was obtained through a 2D CNN model, and the 2D CNN model means a model in which a Convolutional Block Attention Module (CBAM) is added to the ResNet50v2 model.

A spatial attention map of each of the important frame and the non-critical frame may be obtained through a convolutional block attention module (CBAM) calculation.

The Spatial Attention MAP is a value calculated through Convolutional Block Attention Module (CBAM), a technique that improves the performance of a network model by paying attention to a specific vector and performing an operation. More specifically, the attention-focusing technique is used to improve the classification performance of convolutional neural networks by adding weights to important data among data having a hierarchical structure by imitating human recognition processes. This attention-focusing technique has the effect of facilitating the flow of information in the convolutional neural network by distinguishing and learning data containing relatively important information and data containing unimportant information.

For example, spectator information and batter information may be simultaneously included in a frame where the batter is swinging. A MAP maximizing the pixel value of the area where the batter is located is obtained and the pixel value of the area where the spectator is located is minimized. It may mean acquiring one MAP.

A detailed description of the CBAM (Convolutional Block Attention Module) will be described later with reference to FIG. 4 .

The feature vector acquisition module 50 may obtain feature vectors of each of the important frame and the non-important frame. At this time, the Spatial Attention MAP of each of the important frame and the non-critical frame is input as an input value of the LSTM (Long Short Term Memory) model, and the output value of the LSTM (Long Short Term Memory) model is It may be a feature vector or a feature vector of a non-important frame. At this time, the LSTM (Long Short Term Memory) model is a type of recurrent neural network (RNN), and is composed of a cell, an input gate, an output gate, and a forget gate. . At this time, a recurrent neural network (RNN) composed of LSTM (Long Short Term Memory) units is called a LSTM (Long Short Term Memory) model.

The sports activity classification module 60 obtains probability values from feature vectors of important frames and feature vectors of non-important frames, and classifies the sports activities matched with image information constituting the frames only when the probability values are greater than or equal to a preset probability value. can do. At this time, when the probability value is less than a preset probability value, a Long Short Term Memory (LSTM) model may be repeatedly performed.

Fig. 2 is a conceptual diagram showing an important frame extraction module, and Fig. 3 is a conceptual diagram showing how a plurality of frames are multiplied by temporal attention scores through a TAM calculation process.

The important frame extraction module 30 may include a pixel extraction unit 31, a pixel summing unit 32, a pixel pooling operation unit 33, a weight setting unit 34, and a temporal attention score acquisition unit 35. .

The pixel extractor 31 may extract R channel pixels, G channel pixels, and B channel pixels from the frame, and the pixel adder 32 may sum the extracted pixels.

The pixel pooling operation unit 33 may perform MAX-POOLING and AVERAGE-POOLING operations on the sum of pixels. At this time, the size of pixel data may be reduced by performing MAX-POOLING and AVERAGE-POOLING operations.

The weight setting unit 34 sets a weight for each of the maximum pooling (MAX-POOLING) calculation value and the average pooling (AVERAGE-POOLING) calculation value, and sets the weight set for the maximum pooling (MAX-POOLING) and the average pooling (AVERAGE-POOLING) calculation value. POOLING) can be set to 1. At this time, in setting the weight, the user may set a relatively larger weight to the maximum pooling (MAX-POOLING) calculation value, or the user may set a relatively larger weight to the average pooling (AVERAGE-POOLING) calculation value. .

The temporal attentional attention score acquisition unit 35 may obtain a temporal attentional attention score using a value obtained by summing a MAX-POOLING operation value and an AVERAGE-POOLING operation value.

The temporal attention score acquisition unit 35 may include a first fully connected layer, a ReLU activation function execution unit, a second fully connected layer, and a sigmoid activation function execution unit.

In the first fully connected layer, a sum of the maximum pooling (MAX-POOLING) calculation value and the average pooling (AVERAGE-POOLING) calculation value may be input by the pixel pooling calculation unit 33 .

An output value output from the first Fully Connected Layer may be input to the ReLU activation function performer.

An output value output from the ReLU function performer may be input to the second Fully Connected Layer.

An output value output from the second Fully Connected Layer may be input as an input value to the sigmoid activation function execution unit.

A process of calculating the temporal attention score according to an embodiment of the present invention may be expressed by Equation 1 below. This means the TAM (Temporal Attention Module) calculation process.

(Equation 1)

(One)

(2)

(3)

(4)

(5)

은 픽셀 추출부(31)로부터 추출된 R채널, G채널, B채널의 픽셀의 합산 값을 의미할 수 있다.

는 R채널의 픽셀 값,

은 G채널의 픽셀 값,

는 B채널의 픽셀 값을 의미할 수 있다.In Equation (1), I means a specific frame extracted from the frame extraction module 20,

may denote a sum value of pixels of the R channel, the G channel, and the B channel extracted from the pixel extractor 31.

is the pixel value of the R channel,

is the pixel value of the G channel,

may mean a pixel value of the B channel.

는 픽셀 풀링 연산부(33)에 의해 평균풀링(AVERAGE-POOLING) 연산을 수행한 값을 의미할 수 있다.in equation (2)

may mean a value obtained by performing an average pooling operation by the pixel pooling operation unit 33.

는 픽셀 풀링 연산부(33)에 의해 최대풀링(MAX-POOLING)연산을 수행한 값을 의미할 수 있다.in equation (3)

may mean a value obtained by performing a MAX-POOLING operation by the pixel pooling operation unit 33.

은

과

을 합한 값을 의미할 수 있다. a1,a2는 optimal pooling parameter로서, 가중치를 의미할 수 있다. 사용자가

에 더 큰 가중치를 설정하는 경우 a1>a2이며, 사용자가

에 더 큰 가중치를 설정하는 경우 a2>a1일 수 있고, 이 때 a1과 a2의 합은 1로 설정될 수 있다.in equation (4)

silver

class

may mean the sum of a1 and a2 are optimal pooling parameters and may mean weights. user

If you set a larger weight on a1>a2, the user

If a larger weight is set for a2>a1, in this case, the sum of a1 and a2 may be set to 1.

Equation (5) shows a process in which the attention score acquisition unit 35 obtains the temporal attention score using the sum of the MAX-POOLING calculation value and the AVERAGE-POOLING calculation value. expression can mean

은 제1 Fully Connected Layer에 최대풀링(MAX-POOLING) 연산값과 평균풀링(AVERAGE-POOLING) 연산값을 합산한 값이 입력으로 입력되는 것을 의미하며, 이는 다층 신경망 구조의 가중치 벡터를 의미한다. 이 때, Fully Connected Layer는 픽셀 값을 연결하는 완전 결합 층일 수 있으며, 복수개로 마련될 수 있다. 이 때, W1은

의 벡터 형태이고, C는 Fully Connected Layer의 채널을 의미하며, r은 Reduction Ratio를 의미한다.

means that a value obtained by summing the maximum pooling operation value and the average pooling operation value is input to the first Fully Connected Layer as an input, which means a weight vector of a multi-layer neural network structure. At this time, the Fully Connected Layer may be a fully connected layer that connects pixel values, and may be provided in plural. At this time, W1 is

is the vector form of, and C is It means the channel of the Fully Connected Layer, and r means the Reduction Ratio.

는

를 ReLU의 활성화 함수에 입력한 값을 의미할 수 있다. 이 때, ReLU의 활성화 함수는 입력값이 양수인 경우 입력값이 출력값이 되며, 입력값이 음수인 경우 출력값이 0이되는 함수를 의미한다.

Is

may mean a value input to the activation function of ReLU. At this time, the activation function of ReLU means a function in which an input value becomes an output value when the input value is a positive number, and an output value becomes 0 when the input value is a negative number.

는

의 출력 값이 제2 Fully Connected Layer에 입력되는 것을 의미하며, 이는 다층 신경망 구조의 가중치 벡터를 의미한다. 이 때,

의 벡터 형태이고, C는 Fully Connected Layer의 채널을 의미하며, r은 Reduction Ratio를 의미한다.

Is

It means that the output value of is input to the second Fully Connected Layer, which means the weight vector of the multi-layer neural network structure. At this time,

is the vector form of, and C is It means the channel of the Fully Connected Layer, and r means the Reduction Ratio.

는 Sigmoid 함수에 입력한 것을 의미하는 식이며,

로 표현된 Sigmoid 함수는 0에서 1사이값으로 출력된다. 이 때, 출력된 값을 시간적 주의집중 점수라고 한다.

is an expression that means input to the Sigmoid function,

The sigmoid function expressed as is output as a value between 0 and 1. At this time, the output value is called the temporal attention score.

The above is from "S. Liu, X. Ma, H. Wu, Y. Li, An End to End Framework with Adaptive Spatio-Temporal Attention Module for Human Action Recognition, Digital Object Identifier, Vol.8 (2020) 47220- 47231" in detail.

Referring to FIG. 3, a temporal attention module (TAM) calculation process is performed on a frame to obtain a temporal attention score, and a diagram showing a result obtained by multiplying the frame by the temporal attention score.

For example, if the vector value obtained by multiplying the first frame by the temporal attention score is greater than the vector value obtained by multiplying the second frame by the temporal attention score, the first frame is set as an important frame and the second frame is regarded as a non-important frame. means to set 3 shows the result of obtaining a temporal attention score through a TAM (Temporal Attention Module) calculation process and multiplying the temporal attention score by a frame, which means that a frame with a solid border is set as an important frame, and A frame with a dotted line may mean that it is set as a non-critical frame.

4 is a conceptual diagram illustrating a process of obtaining a spatial attention map through a 2D CNN process.

Important frames and non-critical frames multiplied by temporal attention scores may be input to the 2D CNN model. The 2D CNN model may be prepared with a plurality of convolution layers and a convolutional block attention module (CBAM).

The spatial information extraction module 40 may extract spatial information of important frames and non-important frames. In this case, the spatial information may mean a spatial attention map.

The spatial information extraction module 40 may include a convolution layer, a Feauter Map pooling operation unit, a channel attention map acquisition unit, and a spatial attention map generation unit.

If the frame multiplied by the temporal attention score is input to Convolution　Layer, a Feature Map can be output. The feature map is input to the feature map pooling operation unit, and maximum pooling (MAX-POOLING) and average pooling (AVERAGE-POOLING) operations may be performed.

The channel attention map obtaining unit may obtain a channel attention map from an output value of the feature map pooling operation unit.

The spatial attention map acquisition unit may obtain a spatial attention map from the channel attention map.

A process of acquiring a channel attention map and a spatial attention map according to an embodiment of the present invention may be expressed by Equation 2 below. This means a CBAM (Convolution Block Attention Module) calculation process.

(Equation 2)

(One)

(2)

(3)

(4)

를 의미한다.

는

를 채널 축으로 풀링(Pooling)연산을 수행하고, 수행한 결과로 얻어진 벡터를 의미한다.

는

를 채널 축으로 평균풀링(AVERAGE-POOLING)연산을 수행하여 얻어진 벡터를 의미하며,

는

를 채널 축으로 최대풀링(MAX-POOLING)연산을 수행하여 얻어진 벡터를 의미하고, σ는 Sigmoid 함수를 의미하며,

는 채널 주의집중 맵을 의미한다.When the frame multiplied by the temporal attention score in Equation (1) is input to the Convolution Layer, a Feature Map can be obtained, which is

means

Is

It means a vector obtained as a result of performing a pooling operation on the channel axis.

Is

means a vector obtained by performing an average pooling (AVERAGE-POOLING) operation on the channel axis,

Is

means a vector obtained by performing a MAX-POOLING operation on the channel axis, σ means a sigmoid function,

denotes a channel attention map.

는 원소간 곱을 나타내며,

는

를 정제한 Feature Map을 의미한다.in equation (2)

represents the inter-element product,

Is

It means a feature map that has been refined.

₎ 은

을 평균풀링(AVERAGE-POOLING) 연산한 값이며, Maxpool(

₎은

을 최대풀링(MAX-POOLING) 연산한 값을 의미한다.

는 컨벌루션 연산을 의미하고,

은 7x7 컨벌루션 연산을 의미하며,

은 공간적 주의집중 맵(Spatial Attention Map)을 의미한다.In Equation (3), AvgPool(

₎ is

second Average pooling (AVERAGE-POOLING) calculated value, Maxpool (

₎ is

second MAX-POOLING represents the computed value.

means the convolution operation,

means a 7x7 convolution operation,

Means a spatial attention map (Spatial Attention Map).

는

을 정제한 Feature Map을 의미한다. 식(4)에서 얻어진

은 Convolution　Layer에 입력되어 식(3)의 과정이 수행될 수 있다. 즉, 합성곱 신경망의 구조에 따라

의 Feature Map이 새로운 Convolution　Layer에 입력되는 과정이 반복될 수 있는 것이다.Although only two convolution layers are shown in FIG. 4, a plurality of convolution layers may be provided. obtained from equation (4)

Is

It means a feature map that has been refined. obtained from equation (4)

may be input to the Convolution Layer and the process of Equation (3) may be performed. That is, according to the structure of the convolutional neural network

Feature Map of is new The process input to the convolution layer can be repeated.

Calculation of the channel attention map and the spatial attention map can be performed through a software module inserted into one or more data connection sections among the sections between the plurality of convolution layers of the convolutional neural network and the pooling layer for pooling operation. The data connection section between the convolution layer and the pooling layer where map and spatial attention map calculation is performed is not limited to a specific section.

The above is detailed in "S. Woo, J. Park, J. Lee, I. Kweon, CBAM: Convolutional Block Attention Module, Proceedings of the European Conference on Computer Vision (ECCV) (2018) 3-19" .

5 is a conceptual diagram illustrating a process of classifying sports activities after passing through a 2D CNN model.

은 LSTM(Long Short Term Memory)모델의 입력값으로 입력될 수 있다.

may be input as an input value of a Long Short Term Memory (LSTM) model.

An output value of a Long Short Term Memory (LSTM) model may be input to a Multi-Layer Perceptron (MLP). MLP (Multi-Layer Perceptron) is a type of feed-forward artificial neural network composed of at least three layers including an input layer, an output layer, and a hidden layer. network). For training, we use a supervised learning technique called back-propagation. At this time, data that cannot be separated linearly can be distinguished. An output value of MLP (Multi-Layer Perceptron) may be a feature vector.

If the output feature vector of the important frame and the feature vector of the non-important frame are input to the Sigmoid activation function, each probability value can be output. At this time, only when the output probability value is equal to or greater than a preset probability value, it may be classified as a sports activity matched with image information and stored in the database module 10 . If the output probability value is less than the preset probability value, the LSTM process may be repeatedly performed.

6 is a diagram showing a table in which the sports activity classification learning apparatus of the present invention is evaluated through mAP.

In order to evaluate the activity classification performance of the sports activity classification learning device 1 of the present invention, an experiment was conducted using the MLB Youtube Dataset. The MLB Youtube Dataset is a dataset created using videos of MLB 2017 Post Season 20 games in 2017. The dataset includes a total of 5,846 images, and some of the eight sports activities are labeled according to the sports activities within each video: Ball, Strike, Swing, Bunt, Foul, Hit, Hit By Pitch, and In Play. .

In the present invention, in order to search for appropriate hyperparameters, 463 images were extracted from the dataset through Stratified Sampling and used as a verification data set, learning was performed using the remaining 4200 images, and evaluation was performed using 1,183 images. .

mAP (mean average precision) was used as an evaluation scale. mAP is a number often used in fields such as object detection. It is divided by the number of all classes after obtaining the precision for each activity, adding all of them, and is a measure of how well you hit the average on average for all activities.

In FIG. 6, RGB means that only RGB frames are used as inputs, Flow means that only optical flow data is used as inputs, and Two-stream means that RGB frames and optical flows are used as inputs.

Although the proposed method of the present invention uses only RGB frames as input, it shows a higher value than the existing model.

Since the control method of the sports activity classification learning device according to an embodiment of the present invention is performed on substantially the same configuration as the sports activity classification learning device 1 shown in FIG. 1, the sports activity classification learning device 1 and The same reference numerals are given to the same components, and repeated descriptions will be omitted.

7 is a flowchart illustrating a control method of a sports activity classification learning device according to an embodiment of the present invention.

A control method of a sports activity classification learning device that extracts at least one frame from sports activity image information and classifies sports activities, comprising the steps of matching sports activity image information and sports activity classification information and storing them as learning data (100); Extracting at least one frame from sports activity image information stored as data (110), extracting important frames from the extracted frames (120), extracting spatial information of important frames (130), Spatial information is input as an input value of a Long Short Term Memory (LSTM) model (140), feature vectors of important frames are acquired (150), probability values are obtained from feature vectors of important frames (160) and classifying as a sports activity matched with image information constituting an important frame only when the probability value is equal to or greater than a preset probability value ( 170 ).

8 to 9 are flowcharts illustrating a process of acquiring temporal attention points.

Referring to FIG. 8, in step 120 of extracting important frames from the extracted frames, R channel pixels are extracted, G channel pixels are extracted, and B channel pixels are extracted from the frames (200). , summing the extracted pixels (210), performing MAX-POOLING and AVERAGE-POOLING operations on the sum of the pixels (220), MAX-POOLING operation value and A weight is set for each of the average pooling (AVERAGE-POOLING) calculation values, but the sum of the weight set for MAX-POOLING and the weight set for AVERAGE-POOLING is set to 1 (230) and obtaining a temporal attention score by using a value obtained by summing the maximum-pooling operation value and the average pooling operation value (240).

In addition, referring to FIG. 9, in step 240 of obtaining a temporal attention score using the sum of the maximum pooling (MAX-POOLING) calculation value and the average pooling (AVERAGE-POOLING) calculation value, the maximum pooling ( In step 300, the sum of the MAX-POOLING operation value and the AVERAGE-POOLING operation value is input as an input value to the first fully connected layer, and the output value output from the first fully connected layer is used as an input value. Input to the ReLU activation function execution unit (310), the output value output from the ReLU activation function execution unit is input as an input value to the second fully connected layer (320), and the output value output from the second fully connected layer is Sigmoid It may include a step 330 that is input as an input value to the activation function execution unit.

10 is a flowchart illustrating a process of obtaining a spatial attention map.

In the step 130 of extracting spatial information of the important frame, the frame multiplied by the temporal attention score is input to the first convolution layer and outputting a feature map (400), max-pooling the feature map operation and average pooling (AVERAGE-POOLING) operation step (410), channel attention map ( It may include acquiring a Channel Attention Map (420) and acquiring a Spatial Attention Map from the Channel Attention Map (430).

Also, the step of extracting spatial information of non-important frames may be performed in the same manner as the step of extracting spatial information of important frames.

Such a control method of the sports activity classification learning device 1 may be implemented as an application or in the form of program commands that can be executed through various computer components and recorded on a computer-readable recording medium. A computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on a computer-readable recording medium may be those specially designed and configured for the present invention, or may be known and usable to those skilled in the art of computer software.

Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks. -optical media), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like.

Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes such as those produced by a compiler. A hardware device may be configured to act as one or more software modules to perform processing according to the present invention and vice versa.

Although the embodiments of the present invention have been described above, the spirit of the present invention is not limited to the embodiments presented herein, and those skilled in the art who understand the spirit of the present invention may add elements within the scope of the same spirit. However, other embodiments can be easily proposed by means of changes, deletions, additions, etc., but these will also fall within the scope of the present invention.

1: Sports activity classification learning device
10: database module
20: frame extraction module
30: critical frame extraction module
31: pixel extraction unit
32: pixel summation unit
33: pixel pooling operation unit
34: weight setting unit
35: temporal attention score acquisition unit
40: spatial information extraction module
50: feature vector acquisition module
60: sports activity classification module

Claims (10)

A sports activity classification learning device for classifying sports activities by extracting at least one frame from sports activity image information, comprising:
a database module matching the sports activity image information with the sports activity classification information and storing the matching data as learning data;
a frame extraction module extracting at least one frame from the image information stored in the database module;
an important frame extraction module extracting important frames from the extracted frames;
a spatial information extraction module extracting spatial information of the important frame;
a feature vector acquisition module for receiving the spatial information of the important frame as an input value of a Long Short Term Memory (LSTM) model and acquiring a feature vector of the important frame; and
a sports activity classification module that obtains a probability value from the feature vector of the important frame and classifies it into the sports activity matched with the image information constituting the important frame only when the probability value is greater than or equal to a preset probability value; Activity classification learning device.
According to claim 1,
The important frame extraction module,
a pixel extraction unit extracting R channel pixels, G channel pixels, and B channel pixels from the frame;
a pixel summing unit summing the extracted pixels;
a pixel pooling operation unit that performs MAX-POOLING and AVERAGE-POOLING operations on the sum of pixels;
A weight is set for each of the maximum pooling (MAX-POOLING) calculation value and the average pooling (AVERAGE-POOLING) calculation value. a weight setting unit in which the sum of one weight is set to 1; and
and a temporal attentional attention score acquisition unit configured to obtain a temporal attentional attention score using a value obtained by summing the MAX-POOLING calculation value and the AVERAGE-POOLING calculation value.
According to claim 2,
The important frame extraction module,
and extracting, as an important frame, a frame having a relatively large value multiplied by the temporal attention score obtained from the temporal attention score obtaining unit.
According to claim 3,
The temporal attention score acquisition unit,
a first fully connected layer to which a sum of the maximum pooling (MAX-POOLING) calculation value and the average pooling (AVERAGE-POOLING) calculation value is input;
a ReLU activation function performer receiving an output value output from the first Fully Connected Layer as an input value;
a second Fully Connected Layer to which an output value output from the ReLU activation function performer is input as an input value; and
and a sigmoid activation function execution unit to which an output value output from the second Fully Connected Layer is input as an input value.
According to claim 1,
The spatial information extraction module,
a first convolution layer inputting the important frame and outputting a feature map;
a feature map pooling operation unit that performs a MAX-POOLING operation and an AVERAGE-POOLING operation on the feature map;
a channel attention map obtaining unit acquiring a channel attention map from an output value of the feature map pooling operation unit; and
and a spatial attention map acquiring unit acquiring a spatial attention map from the channel attention map.
A control method of a sports activity classification learning device for classifying sports activities by extracting at least one frame from sports activity image information, comprising:
Matching the sports activity image information and the sports activity classification information and storing them as learning data;
Extracting at least one frame from the sports activity image information stored as the learning data;
Extracting important frames from the extracted frames;
Extract spatial information of the important frame;
Spatial information of the important frame is input as an input value of a Long Short Term Memory (LSTM) model, and a feature vector of the important frame is obtained;
Obtaining a probability value from the feature vector of the important frame, and classifying it as the sports activity matched with the image information constituting the important frame only when the probability value is equal to or greater than a preset probability value. control method.
According to claim 6,
Extracting important frames from the extracted frames,
extracting R channel pixels from the frame, extracting G channel pixels, extracting B channel pixels, and summing the extracted pixels;
Perform MAX-POOLING and AVERAGE-POOLING operations on the sum of pixels,
A weight is set for each of the maximum pooling (MAX-POOLING) calculation value and the average pooling (AVERAGE-POOLING) calculation value. The sum of one weight is set to 1,
The control method of the sports activity classification learning apparatus comprising obtaining a temporal attention score by using a sum of the maximum pooling (MAX-POOLING) calculation value and the average pooling (AVERAGE-POOLING) calculation value.
According to claim 7,
Obtaining a temporal attention score using the sum of the MAX-POOLING operation value and the AVERAGE-POOLING operation value,
A value obtained by summing the MAX-POOLING calculation value and the AVERAGE-POOLING calculation value is input as an input value to the first Fully Connected Layer,
The output value output from the first Fully Connected Layer is input to the ReLU activation function performing unit as an input value,
An output value output from the ReLU activation function performing unit is input as an input value to a second Fully Connected Layer;
and inputting an output value output from the second Fully Connected Layer as an input value to a sigmoid activation function performing unit.
According to claim 6,
Extracting the spatial information of the important frame,
A frame multiplied by the temporal attention score is input to a first convolution layer to output a feature map,
Perform MAX-POOLING and AVERAGE-POOLING operations on the Feature Map,
Obtaining a channel attention map using a value obtained by performing MAX-POOLING and AVERAGE-POOLING operations on the feature map;
A control method of a sports activity classification learning apparatus comprising obtaining a spatial attention map from the channel attention map.
A computer-readable storage medium on which a computer program for performing the control method of the sports activity classification learning device according to any one of claims 6 to 9 is recorded.

Images