KR102702069B1 - Method of controlling sports activity classification learning apparatus, computer readable medium and apparatus for performing the method - Google Patents

Abstract

The present invention provides a sports activity classification learning device including: a database module for matching sports activity video information and the sports activity classification information and storing them as learning data; a frame extraction module for extracting frames; an important frame extraction module for extracting important frames from the extracted frames; a spatial information extraction module for extracting spatial information of important frames; a feature vector acquisition module for acquiring feature vectors of important frames; and a sports activity classification module for classifying them as sports activities.

Description

METHOD OF CONTROLLING SPORTS ACTIVITY CLASSIFICATION LEARNING APPARATUS, COMPUTER READABLE MEDIUM AND APPARATUS FOR PERFORMING THE METHOD

The present invention relates to a control method for a sports activity classification learning device, a recording medium and a device for performing the same, and more specifically, to a control method for a sports activity classification learning device that classifies sports activities by inputting at least one frame from sports activity image information, and a recording medium and a device for performing the same.

Activity recognition technology is a technology that recognizes what activities are occurring and classifies the recognized activities.

Activity recognition can be done with a sensor-based method. Sensor-based activity recognition uses sensors such as accelerometers and gyroscopes. However, in sports games, athletes are prohibited from wearing any electromagnetic equipment including sensors such as accelerometers and gyroscopes. Therefore, there is a limitation that sensor-based activity recognition methods are difficult to use in the field of sports game activity recognition.

As a result, various deep learning techniques such as deep neural networks (DNNs), convolutional neural networks (CNNs), deep belief networks (DBNs), and recurrent neural networks (RNNs) are required to be applied to video information of sports games to recognize athletes' sports activities and classify recognized sports activities.

The technical problem to be solved by the present invention is to provide a control method for a sports activity classification learning device that extracts frames from video information of a sports game to obtain a temporal attention score and a spatial attention map to classify sports activities, and a recording medium and device for performing the same.

One aspect of the present invention is a sports activity classification learning device that extracts at least one frame from sports activity image information and classifies sports activities, the device including: a database module that matches the sports activity image information with the sports activity classification information and stores the matched information as learning data; a frame extraction module that extracts at least one frame from the image information stored in the database module; an important frame extraction module that extracts an important frame from the extracted frame; a spatial information extraction module that extracts spatial information of the important frame; a feature vector acquisition module that inputs the spatial information of the important frame as an input value of an LSTM (Long Short Term Memory) model and acquires a feature vector of the important frame; and a sports activity classification module that acquires a probability value from the feature vector of the important frame and classifies the sport activity as matching the image information constituting the important frame only when the probability value is greater than or equal to a preset probability value.

In addition, the important frame extraction module may include a pixel extraction unit that extracts pixels of an R channel from the frame, pixels of a G channel, and pixels of a B channel; a pixel summing unit that sums the extracted pixels; a pixel pooling operation unit that performs MAX-POOLING and AVERAGE-POOLING operations on the summed pixel values; a weight setting unit that sets weights for each of the MAX-POOLING operation values and the AVERAGE-POOLING operation values, wherein the sum of the weights set for the MAX-POOLING operation value and the weights set for the AVERAGE-POOLING operation value is set to 1; and a temporal attention score obtaining unit that obtains a temporal attention score by using a value obtained by summing the MAX-POOLING operation value and the AVERAGE-POOLING operation value.

In addition, the important frame extraction module may include extracting a frame having a relatively large value multiplied by a temporal attention score obtained from the temporal attention score obtaining unit as an important frame.

In addition, the temporal attention score acquisition unit may include a first Fully Connected Layer into which a sum of the MAX-POOLING operation value and the AVERAGE-POOLING operation value is input; a ReLU activation function execution unit into which an output value output from the first Fully Connected Layer is input as an input value; a second Fully Connected Layer into which an output value output from the ReLU activation function execution unit is input as an input value; and a Sigmoid activation function execution unit into which an output value output from the second Fully Connected Layer is input as an input value.

In addition, the spatial information extraction module may include a first convolution layer for inputting the important frame and outputting a feature map; a feature map pooling operation unit for performing a max pooling operation and an average pooling operation on the feature map; a channel attention map acquisition unit for acquiring a channel attention map from an output value of the feature map pooling operation unit; and a spatial attention map acquisition unit for acquiring a spatial attention map from the channel attention map.

Another aspect of the present invention is a control method of a sports activity classification learning device that extracts at least one frame from sports activity image information and classifies a sports activity, the control method including matching the sports activity image information and the sports activity classification information and storing the matching as learning data, extracting at least one frame from the sports activity image information stored as the learning data, extracting an important frame from the extracted frame, extracting spatial information of the important frame, inputting the spatial information of the important frame as an input value of an LSTM (Long Short Term Memory) model, obtaining a feature vector of the important frame, obtaining a probability value from the feature vector of the frame, and classifying the sports activity as matching the image information constituting the important frame only when the probability value is greater than or equal to a preset probability value.

In addition, extracting important frames from the extracted frames may include extracting pixels of an R channel from the frames, extracting pixels of a G channel, extracting pixels of a B channel, summing the extracted pixels, performing MAX-POOLING and AVERAGE-POOLING operations on the summed values of the pixels, setting weights for each of the MAX-POOLING operation values and the AVERAGE-POOLING operation values, wherein the sum of the weights set for the MAX-POOLING operation value and the weights set for the AVERAGE-POOLING operation value is set to 1, and obtaining a temporal attention score by using a value obtained by summing the MAX-POOLING operation value and the AVERAGE-POOLING operation value.

In addition, obtaining a temporal attention score by using the sum of the MAX-POOLING operation value and the AVERAGE-POOLING operation value may include inputting the sum of the MAX-POOLING operation value and the AVERAGE-POOLING operation value as an input value to a first Fully Connected Layer, inputting the output value output from the first Fully Connected Layer as an input value to a ReLU activation function execution unit, inputting the output value output from the ReLU activation function execution unit as an input value to a second Fully Connected Layer, and inputting the output value output from the second Fully Connected Layer as an input value to a Sigmoid activation function execution unit.

In addition, extracting spatial information of the important frame may include inputting a frame multiplied by the temporal attention score into a first convolution layer to output a feature map, performing a MAX-POOLING operation and an AVERAGE-POOLING operation on the feature map, obtaining a channel attention map using values obtained by performing the MAX-POOLING operation and the AVERAGE-POOLING operation on the feature map, and obtaining a spatial attention map from the channel attention map.

In another aspect of the present invention, a computer program for performing a control method of a sports activity classification learning device can be recorded on a computer-readable storage medium.

According to one aspect of the present invention described above, by providing a control method of a sports activity classification learning device that classifies sports activities, a recording medium and a device for performing the same, sports activities can be classified by considering both a temporal attention score and a spatial attention map.

Figure 1 is a conceptual diagram illustrating a sports activity classification learning device according to one embodiment of the present invention.
Figure 2 is a conceptual diagram showing the important frame extraction module.
Figure 3 is a conceptual diagram showing how temporal attention scores are multiplied across multiple frames through the TAM calculation process.
Figure 4 is a conceptual diagram showing the process of obtaining a spatial attention map through a 2D CNN model.
Figure 5 is a conceptual diagram showing the process of classifying sports activities after passing through a 2D CNN model.
Figure 6 is a diagram showing a table evaluating the sports activity classification learning device of the present invention through mAP.
Figure 7 is a flowchart showing a control method of a sports activity classification learning device according to one embodiment of the present invention.
Figures 8 and 9 are flowcharts showing the process of obtaining temporal attention scores.
Figure 10 is a flowchart showing the process of obtaining a spatial attention map.

The detailed description of the invention set forth below refers to the accompanying drawings which illustrate, by way of example, specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that the various embodiments of the invention, while different from one another, are not necessarily mutually exclusive. For example, specific shapes, structures, and features described herein may be implemented in other embodiments without departing from the spirit and scope of the invention. It should also be understood that the positions or arrangements of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the invention. Accordingly, the detailed description set forth below is not to be taken in a limiting sense, and the scope of the invention is defined only by the appended claims, along with the full scope of equivalents to which such claims are entitled, if properly so described. Like reference numerals in the drawings designate the same or similar functionality throughout the several aspects.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the drawings.

Figure 1 is a conceptual diagram illustrating a sports activity classification learning device according to one embodiment of the present invention.

A sports activity classification learning device (1) may include a database module (10), a frame extraction module (20), an important frame extraction module (30), a spatial information extraction module (40), a feature vector acquisition module (50), and a sports activity classification module (60).

A sports activity classification learning device (1) may be a device that extracts at least one frame from sports activity video information and learns to classify what kind of sports activity is occurring from the extracted frame. At this time, the sports activity may be an activity in which a batter is swinging in a baseball game, an activity in which a batter is hitting a home run or a strike, etc., and may include an activity in which a player is scoring a goal in a soccer game, an activity in which a player is shooting, an activity in which a corner kick or a free kick is being taken, an activity in which a foul is being committed, an activity in which a foul is being committed, an activity in which a penalty kick is being committed, etc. However, examples of sports activities have been given, but are not limited thereto.

The database module (10) can match sports activity video information and sports activity classification information and store them as learning data. For example, if the sports activity video information is video information of a batter swinging in a baseball game, the sports activity classification information can be matched and stored as 'swing', and if the sports activity video information is video information of a batter hitting a home run in a baseball game, the sports activity classification information can be matched and stored as 'home run'.

The frame extraction module (20) can extract at least one frame from the image information stored in the database module (10). For example, in a baseball game, at least one continuous frame can be extracted at a preset time interval from the image information of a batter hitting a home run, or at least one discontinuous frame can be extracted.

The important frame extraction module (30) can calculate a temporal attention score from the pixel value of the frame and multiply the temporal attention score by the frame. The method for calculating the temporal attention score can calculate the temporal attention score through the Temporal Attention Module (TAM). At this time, the Temporal Attention Module (TAM) is a method for extracting important frames and non-important frames by calculating the importance between multiple frames.

For example, if the value of the second frame multiplied by the temporal attention score is greater than that of the first frame multiplied by the temporal attention score, the second frame may be extracted as an important frame, and the first frame may be extracted as a non-important frame.

A detailed description of the TAM (Temporal Attention Module) will be given in detail later in Figures 2 and 3.

The spatial information extraction module (40) can obtain a spatial attention map of important and non-important frames multiplied by a temporal attention score.

The present invention obtains a spatial attention map through a 2D CNN model, and the 2D CNN model refers to a model in which a CBAM (Convolutional Block Attention Module) is added to the ResNet50v2 model.

Spatial attention maps for each important and unimportant frame can be obtained through CBAM (Convolutional Block Attention Module) calculation.

The spatial attention map (Spatial Attention MAP) is a value produced by the Convolutional Block Attention Module (CBAM), a technique that improves the performance of the network model by performing operations by focusing on specific vectors. More specifically, the attention technique is used to improve the classification performance of the convolutional neural network by adding weights to important data among data with a hierarchical structure by imitating the human recognition process. This attention technique has the effect of facilitating the flow of information within the convolutional neural network by distinguishing and learning data containing relatively important information from data containing unimportant information.

For example, a frame in which a batter is swinging may include both audience information and batter information, which may mean obtaining a MAP that maximizes the pixel value of the area where the batter is located and obtaining a MAP that minimizes the pixel value of the area where the spectator is located.

A detailed description of the CBAM (Convolutional Block Attention Module) is provided later in Fig. 4.

The feature vector acquisition module (50) can acquire feature vectors of each of the important and non-important frames. At this time, the spatial attention maps of each of the important and non-important frames are input as input values of the LSTM (Long Short Term Memory) model, and the output values of the LSTM (Long Short Term Memory) model can be the feature vectors of the important and non-important frames. At this time, the LSTM (Long Short Term Memory) model is a type of recurrent neural network (RNN) and is composed of a cell, an input gate, an output gate, and a forget gate. At this time, a recurrent neural network (RNN) composed of LSTM (Long Short Term Memory) units is called a LSTM (Long Short Term Memory) model.

The sports activity classification module (60) obtains probability values from the feature vectors of important frames and the feature vectors of unimportant frames, and can classify the frame into a sports activity matching the image information only when the probability value is greater than or equal to a preset probability value. In this case, when the probability value is less than the preset probability value, the LSTM (Long Short Term Memory) model can be repeatedly performed.

Figure 2 is a conceptual diagram showing a key frame extraction module, and Figure 3 is a conceptual diagram showing the temporal attention scores multiplied to multiple frames through the TAM calculation process.

The important frame extraction module (30) may include a pixel extraction unit (31), a pixel summation unit (32), a pixel pooling operation unit (33), a weight setting unit (34), and a temporal attention score acquisition unit (35).

The pixel extraction unit (31) can extract pixels of the R channel from the frame, pixels of the G channel, and pixels of the B channel, and the pixel summing unit (32) can sum the extracted pixels.

The pixel pooling operation unit (33) can perform maximum pooling (MAX-POOLING) and average pooling (AVERAGE-POOLING) operations on the sum of pixels. At this time, by performing maximum pooling (MAX-POOLING) and average pooling (AVERAGE-POOLING) operations, the size of pixel data can be reduced.

The weight setting unit (34) sets weights for each of the MAX-POOLING operation value and the AVERAGE-POOLING operation value, but the sum of the weights set for MAX-POOLING and the weights set for AVERAGE-POOLING can be set to 1. At this time, when setting the weights, the user can set a relatively larger weight for the MAX-POOLING operation value, or the user can set a relatively larger weight for the AVERAGE-POOLING operation value.

The temporal attention score acquisition unit (35) can acquire the temporal attention score by using the value obtained by adding the maximum pooling (MAX-POOLING) operation value and the average pooling (AVERAGE-POOLING) operation value.

The temporal attention score acquisition unit (35) can be composed of a first fully connected layer, a ReLU activation function execution unit, a second fully connected layer, and a sigmoid activation function execution unit.

The first fully connected layer can input a value that is the sum of the maximum pooling (MAX-POOLING) operation value and the average pooling (AVERAGE-POOLING) operation value by the pixel pooling operation unit (33).

The ReLU activation function execution unit can input the output value output from the first fully connected layer.

The second fully connected layer can receive the output value from the ReLU function execution unit.

The sigmoid activation function execution section can input the output value from the second fully connected layer as an input value.

The process of calculating a temporal attention score according to one embodiment of the present invention can be expressed by the following mathematical expression 1. This refers to the TAM (Temporal Attention Module) calculation process.

(Mathematical formula 1)

(1)

(2)

(3)

(4)

(5)

In equation (1), I means a specific frame extracted from the frame extraction module (20), may mean the sum of the pixels of the R channel, G channel, and B channel extracted from the pixel extraction unit (31). is the pixel value of the R channel, is the pixel value of the G channel, can mean the pixel value of the B channel.

In equation (2) may mean a value obtained by performing an average pooling operation by a pixel pooling operation unit (33).

In equation (3) may mean a value obtained by performing a maximum pooling operation by a pixel pooling operation unit (33).

In equation (4) silver class It can mean the sum of a1 and a2. a1 and a2 are optimal pooling parameters and can mean weights. The user If you set a larger weight to a1>a2, the user If a larger weight is set, a2>a1, in which case the sum of a1 and a2 can be set to 1.

Equation (5) may mean an equation that represents the process in which the attention score acquisition unit (35) acquires a temporal attention score by using the value obtained by adding the maximum pooling (MAX-POOLING) operation value and the average pooling (AVERAGE-POOLING) operation value.

It means that the value of the sum of the MAX-POOLING operation value and the AVERAGE-POOLING operation value is input to the first Fully Connected Layer, which means the weight vector of the multilayer neural network structure. At this time, the Fully Connected Layer can be a fully connected layer that connects pixel values, and can be provided in multiple layers. At this time, W1 is is a vector form, and C is It refers to the channel of the Fully Connected Layer, and r refers to the Reduction Ratio.

Is It can mean the value input to the activation function of ReLU. At this time, the activation function of ReLU means a function in which the input value becomes the output value when the input value is positive, and the output value becomes 0 when the input value is negative.

Is This means that the output value is input to the second fully connected layer, which means the weight vector of the multilayer neural network structure. At this time, is a vector form, and C is It refers to the channel of the Fully Connected Layer, and r refers to the Reduction Ratio.

is an expression that means what is input to the Sigmoid function, The Sigmoid function, expressed as , outputs a value between 0 and 1. At this time, the output value is called the temporal attention score.

The above is described in detail in "S. Liu, X. Ma, H. Wu, Y. Li, An End to End Framework with Adaptive Spatio-Temporal Attention Module for Human Action Recognition, Digital Object Identifier, Vol.8 (2020) 47220-47231".

Referring to Figure 3, this is a diagram showing the result of obtaining a temporal attention score by going through the Temporal Attention Module (TAM) calculation process for a frame and multiplying the temporal attention score by the frame.

For example, if the vector value obtained by multiplying the temporal attention score of the first frame is greater than the vector value obtained by multiplying the temporal attention score of the second frame, it means that the first frame is set as an important frame and the second frame is set as a non-important frame. Fig. 3 shows the result of obtaining a temporal attention score through the TAM (Temporal Attention Module) calculation process and multiplying the temporal attention score by the frame. This can mean that a frame with a solid border is set as an important frame, and a frame with a dotted border is set as a non-important frame.

Figure 4 is a conceptual diagram showing the process of obtaining a spatial attention map through a 2D CNN process.

Important and unimportant frames multiplied by temporal attention scores can be input to a 2D CNN model. The 2D CNN model can be constructed with multiple convolutional layers and a CBAM (Convolutional Block Attention Module).

The spatial information extraction module (40) can extract spatial information of important frames and non-important frames. At this time, spatial information may mean a spatial attention map.

The spatial information extraction module (40) may include a convolution layer, a factor map pooling operation unit, a channel attention map acquisition unit, and a spatial attention map generation unit.

When a frame multiplied by a temporal attention score is input to a convolution layer, a feature map can be output. The feature map can be input to a feature map pooling operation unit, and a maximum pooling (MAX-POOLING) operation and an average pooling (AVERAGE-POOLING) operation can be performed.

The channel attention map acquisition unit can acquire a channel attention map from the output value of the feature map pooling operation unit.

A spatial attention map acquisition unit can acquire a spatial attention map from a channel attention map.

The process of obtaining a channel attention map and a spatial attention map according to one embodiment of the present invention can be expressed by the following mathematical expression 2. This means a CBAM (Convolution Block Attention Module) calculation process.

(Mathematical formula 2)

(1)

(2)

(3)

(4)

When the frame multiplied by the temporal attention score in Equation (1) is input to the Convolution Layer, a Feature Map can be obtained, which is It means. Is It means a vector obtained as a result of performing a pooling operation on the channel axis. Is It refers to a vector obtained by performing an average pooling operation on the channel axis. Is It means a vector obtained by performing a MAX-POOLING operation on the channel axis, and σ means a Sigmoid function. stands for channel attention map.

In equation (2) represents the product of elements, Is It refers to a refined Feature Map.

In Equation (3), AvgPool( ₎ silver second This is the value calculated by AVERAGE-POOLING, and Maxpool( ₎ silver second MAX-POOLING It means the calculated value. means convolution operation, stands for 7x7 convolution operation, stands for Spatial Attention Map.

In Fig. 4, only two convolution layers are depicted, but multiple convolution layers can be provided. Obtained from Equation (4) Is It means the refined feature map obtained from equation (4). is input to the Convolution Layer and the process of Equation (3) can be performed. That is, according to the structure of the convolutional neural network. The Feature Map of the new The process of inputting to the Convolution Layer can be repeated.

The calculation of the channel attention map and the spatial attention map can be performed through a software module inserted into one or more data connection sections among the sections between multiple convolution layers of the convolution neural network and the pooling layer for the pooling operation, and the data connection section between the convolution layer and the pooling layer where the channel attention map and the spatial attention map calculation are performed is not limited to a specific section.

The above is described in detail in "S. Woo, J. Park, J. Lee, I. Kweon, CBAM: Convolutional Block Attention Module, Proceedings of the European Conference on Computer Vision (ECCV) (2018) 3-19 ".

Figure 5 is a conceptual diagram showing the process of classifying sports activities after passing through a 2D CNN model.

can be input as an input value for the LSTM (Long Short Term Memory) model.

The output of the LSTM (Long Short Term Memory) model can be input to the MLP (Multi-Layer Perceptron). The MLP (Multi-Layer Perceptron) is a type of feed-forward artificial neural network consisting of at least three layers, including an input layer, an output layer, and a hidden layer. It uses a supervised learning technique called back-propagation for training. At this time, data that cannot be separated linearly can be distinguished. The output of the MLP (Multi-Layer Perceptron) can be a feature vector.

When the feature vector of the output important frame and the feature vector of the unimportant frame are input to the sigmoid activation function, the probability value of each can be output. At this time, only when the output probability value is greater than the preset probability value, it can be classified as a sports activity stored by matching with the image information in the database module (10). If the output probability value is less than the preset probability value, the LSTM process can be repeatedly performed.

Figure 6 is a diagram showing a table evaluating the sports activity classification learning device of the present invention through mAP.

In order to evaluate the activity classification performance of the sports activity classification learning device (1) of the present invention, an experiment was conducted using the MLB Youtube Dataset. The MLB Youtube Dataset is a dataset created using videos of MLB 2017 Post Season 20 games in 2017. The dataset contains a total of 5,846 videos, and some of the eight sports activities, Ball, Strike, Swing, Bunt, Foul, Hit, Hit By Pitch, and In Play, are labeled according to the sports activities in each video.

In the present invention, for appropriate hyperparameter search, 463 images were extracted from the dataset through Stratified Sampling and used as a verification dataset, the remaining 4,200 images were used for learning, and 1,183 images were used for evaluation.

mAP (mean Average Precision) was used as the evaluation metric. mAP is a frequently used metric in fields such as Object Detection. It calculates the precision for each activity, adds them all up, and then divides them by the total number of classes. It is a metric that shows how accurate the average is for all activities.

In Fig. 6, RGB means that only RGB frames were used as input, Flow means that only optical flow data was used as input, and Two-stream means that both RGB frames and optical flow were used as input.

The Proposed Method of the present invention shows higher values than existing models even though it uses only RGB frames as input.

The control method of the sports activity classification learning device according to one embodiment of the invention is performed on a configuration substantially identical to that of the sports activity classification learning device (1) illustrated in FIG. 1, and therefore the same drawing reference numerals are assigned to the same components as those of the sports activity classification learning device (1) illustrated in FIG. 1, and repetitive descriptions are omitted.

Figure 7 is a flowchart showing a control method of a sports activity classification learning device according to one embodiment of the present invention.

A control method of a sports activity classification learning device that extracts at least one frame from sports activity image information and classifies sports activities, the control method may include a step (100) of matching sports activity image information and sports activity classification information and storing them as learning data, a step (110) of extracting at least one frame from the sports activity image information stored as learning data, a step (120) of extracting important frames from the extracted frames, a step (130) of extracting spatial information of the important frames, a step (140) of inputting the spatial information of the important frames as input values of an LSTM (Long Short Term Memory) model, a step (150) of obtaining a feature vector of the important frames, a step (160) of obtaining a probability value from the feature vector of the important frames, and a step (170) of classifying the sports activity as matching the image information constituting the important frames only when the probability value is greater than or equal to a preset probability value.

Figures 8 and 9 are flowcharts showing the process of obtaining temporal attention scores.

Referring to FIG. 8, the step (120) of extracting important frames from the extracted frames may include the steps (200) of extracting pixels of the R channel from the frame, pixels of the G channel, and pixels of the B channel, the step (210) of summing the extracted pixels, the step (220) of performing MAX-POOLING and AVERAGE-POOLING operations on the summed pixel value, the step (230) of setting weights for each of the MAX-POOLING operation value and the AVERAGE-POOLING operation value, but setting the sum of the weights set for MAX-POOLING and the weights set for AVERAGE-POOLING to 1, and the step (240) of obtaining a temporal attention score using the sum of the MAX-POOLING operation value and the AVERAGE-POOLING operation value.

In addition, referring to FIG. 9, the step (240) of obtaining a temporal attention score by using the sum of the MAX-POOLING operation value and the AVERAGE-POOLING operation value may include the step (300) of inputting the sum of the MAX-POOLING operation value and the AVERAGE-POOLING operation value as an input value to the first fully connected layer, the step (310) of inputting the output value output from the first fully connected layer as an input value to a ReLU activation function execution unit, the step (320) of inputting the output value output from the ReLU activation function execution unit as an input value to the second fully connected layer, and the step (330) of inputting the output value output from the second fully connected layer as an input value to a Sigmoid activation function execution unit.

Figure 10 is a flowchart showing the process of obtaining a spatial attention map.

The step (130) of extracting spatial information of an important frame may include a step (400) of inputting a frame multiplied by a temporal attention score into a first convolution layer to output a feature map, a step (410) of performing a MAX-POOLING operation and an AVERAGE-POOLING operation on the feature map, a step (420) of obtaining a channel attention map using the values obtained by performing the MAX-POOLING operation and the AVERAGE-POOLING operation on the feature map, and a step (430) of obtaining a spatial attention map from the channel attention map.

Additionally, the step of extracting spatial information of non-important frames can be performed in the same manner as the step of extracting spatial information of important frames.

The control method of the sports activity classification learning device (1) of this kind may be implemented as an application or may be implemented in the form of program commands that can be executed through various computer components and may be recorded on a computer-readable recording medium. The computer-readable recording medium may include program commands, data files, data structures, etc., singly or in combination. The program commands recorded on the computer-readable recording medium may be those specifically designed and configured for the present invention, or may be those known to and usable by those skilled in the art of computer software.

Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory.

Examples of program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. A hardware device may be configured to operate as one or more software modules to perform processing according to the present invention, and vice versa.

Although the embodiments of the present invention have been described above, the spirit of the present invention is not limited to the embodiments presented in this specification, and those skilled in the art who understand the spirit of the present invention will be able to easily propose other embodiments by adding, changing, deleting, or adding components within the scope of the same spirit, but this will also be considered to fall within the spirit of the present invention.

1: Sports activity classification learning device
10: Database Module
20: Frame Extraction Module
30: Critical Frame Extraction Module
31: Pixel Extraction Section
32: Pixel summation unit
33: Pixel pooling operation unit
34: Weight setting section
35: Temporal Attention Score Acquisition Section
40: Spatial information extraction module
50: Feature vector acquisition module
60: Sports Activity Classification Module

Claims (10)

A sports activity classification learning device that extracts at least one frame from sports activity video information and classifies sports activities,
A database module that matches the above sports activity video information and sports activity classification information and stores them as learning data;
A frame extraction module for extracting at least one frame from image information stored in the above database module;
An important frame extraction module that extracts important frames from the above extracted frames;
A spatial information extraction module for extracting spatial information of the above important frame;
A feature vector acquisition module in which spatial information of the above important frame is input as an input value of an LSTM (Long Short Term Memory) model and a feature vector of the above important frame is acquired; and
A sports activity classification module that obtains a probability value from a feature vector of the above important frame and classifies the sports activity as matching the image information constituting the above important frame only when the probability value is greater than or equal to a preset probability value;
The above important frame extraction module is,
A pixel extraction unit that extracts pixels of the R channel, pixels of the G channel, and pixels of the B channel from the above frame;
A pixel summing unit that sums the above extracted pixels;
A pixel pooling operation unit that performs MAX-POOLING and AVERAGE-POOLING operations on the sum of pixels;
A weight setting unit in which weights are set for each of the above maximum pooling (MAX-POOLING) operation value and the average pooling (AVERAGE-POOLING) operation value, and the sum of the weights set for the maximum pooling (MAX-POOLING) and the weights set for the average pooling (AVERAGE-POOLING) is set to 1; and
A sports activity classification learning device including a temporal attention score acquisition unit that acquires a temporal attention score by using the value obtained by adding the MAX-POOLING operation value and the AVERAGE-POOLING operation value.
delete In paragraph 1,
The above important frame extraction module is,
A sports activity classification learning device including extracting a frame having a relatively large value multiplied by a temporal attention score obtained from the temporal attention score obtaining unit as an important frame.
In the third paragraph,
The above temporal attention focus score acquisition section is,
The first fully connected layer inputs the sum of the MAX-POOLING operation value and the AVERAGE-POOLING operation value;
A ReLU activation function execution unit in which the output value output from the first Fully Connected Layer is input as an input value;
A second fully connected layer in which the output value output from the above ReLU activation function performing unit is input as an input value; and
A sports activity classification learning device including a sigmoid activation function performing unit in which the output value output from the second fully connected layer is input as an input value.
In paragraph 1,
The above spatial information extraction module,
The first Convolution Layer, which inputs the above important frame and outputs a Feature Map;
Feature map pooling operation unit that performs MAX-POOLING operation and AVERAGE-POOLING operation on the above feature map;
A channel attention map acquisition unit that acquires a channel attention map from the output value of the above feature map pooling operation unit; and
A sports activity classification learning device including a spatial attention map acquisition unit that acquires a spatial attention map from the above channel attention map.
A control method of a sports activity classification learning device that classifies sports activities by extracting at least one frame from sports activity video information,
Match the above sports activity video information and sports activity classification information and save them as learning data.
Extracting at least one frame from the sports activity video information stored as the above learning data,
Extract important frames from the above extracted frames,
Extract spatial information of the above important frames,
The spatial information of the above important frame is input as an input value of the LSTM (Long Short Term Memory) model, and the feature vector of the above important frame is obtained.
Including obtaining a probability value from the feature vector of the above important frame, and classifying the sport activity as matching the image information constituting the above important frame only when the probability value is greater than or equal to a preset probability value,
Extracting important frames from the above extracted frames is as follows:
Extract the pixels of the R channel from the above frame, extract the pixels of the G channel, extract the pixels of the B channel, and combine the extracted pixels.
Perform MAX-POOLING and AVERAGE-POOLING operations on the sum of pixels.
A weight is set for each of the above maximum pooling (MAX-POOLING) operation values and average pooling (AVERAGE-POOLING) operation values, but the sum of the weight set for the maximum pooling (MAX-POOLING) and the weight set for the average pooling (AVERAGE-POOLING) is set to 1.
A control method for a sports activity classification learning device, which includes obtaining a temporal attention score by using a value obtained by adding the MAX-POOLING operation value and the AVERAGE-POOLING operation value.
delete In paragraph 6,
The temporal attention score is obtained by using the sum of the MAX-POOLING operation value and the AVERAGE-POOLING operation value.
The sum of the above MAX-POOLING operation value and the AVERAGE-POOLING operation value is input to the first Fully Connected Layer as an input value.
The output value from the first Fully Connected Layer is input as an input value to the ReLU activation function execution unit.
The output value from the above ReLU activation function execution section is input as an input value to the second fully connected layer.
A control method for a sports activity classification learning device, comprising inputting an output value output from the second fully connected layer as an input value to a sigmoid activation function performing unit.
In paragraph 6,
Extracting spatial information of the above important frames is:
The frame multiplied by the above temporal attention score is input to the first convolution layer to output a feature map.
MAX-POOLING and AVERAGE-POOLING operations are performed on the above feature map.
A channel attention map is obtained by using the values obtained by performing the MAX-POOLING operation and the AVERAGE-POOLING operation of the above feature map.
A control method for a sports activity classification learning device, comprising obtaining a spatial attention map from the above channel attention map.
A computer-readable storage medium having recorded thereon a computer program for performing a control method of the sports activity classification learning device according to Article 6.

Images