KR20170025535A - Method of modeling a video-based interactive activity using the skeleton posture datset - Google Patents

Abstract

The present invention relates to a video-based interactive modeling method using a set of skeleton attitude data, wherein a method for modeling an interactive activity according to an embodiment includes the steps of receiving a data set of a two-dimensional skeleton position extracted from a video, Calculating tracking coordinates including the motion velocity and the motion direction of the object from the position coordinates of the calculated object, calculating a distance between the mutual zone corresponding to the object and the calculated tracking characteristics Calculating features from the data set of skeletons for the determined cross-object, and modeling the calculated features into topics of single human actions and inter-group activities. ≪ RTI ID = 0.0 > do.

Description

METHOD OF MODELING A VIDEO-BASED INTERACTIVE ACTIVITY USING THE SKELETON POSTURE DATSET < RTI ID = 0.0 >

And a technique for generating a mutual activity model based on a video using a skeleton attitude data set.

Although computer vision and artificial intelligence communities have received more attention in recent decades, human perception remains a challenging problem due to changes in appearance, mutual occlusion, and the interaction of multiple objects.

While previous techniques have attempted to perceive human activity using the behavior of human components as an input feature, recent technologies have focused on techniques for collecting low-level features. For example, due to the limitations of image processing techniques, there is a tendency to concentrate on techniques for collecting low-level features such as spatial spatial-temporal features instead of techniques for representing the human body, such as a skeleton.

US Patent No. 7,366,645 (entitled " Method of Recognition of Human Motion, Vector Sequences and Speech)

Y. Yang and D. Ramanan, "Articulated human detection with flexible mixtures of parts ", Pattern Analysis and Machine Learning, IEEE Transaction on, vol. 36, no. 9, pp. 1775-1788, Sept 2014. W. Yang, Y. Wang, and G. Mori, "Recognizing Human Actions from Still Images with Latent Poses," in Computer Vision and Pattern Recognition, 2010 International Conference on. San Francisco, USA, IEEE, 2010, pp. 2030-2037

A technique for representing human interactive actions from video captured from surveillance cameras is presented.

Within a group or between groups, a technique for modeling interactions between mutual human objects is presented.

By omitting non-interacted objects, the goal is to reduce the computational cost.

The aim is to improve the quality of the training data set of the function because the singular values are not included in the data set.

The goal is to improve classification accuracy in distinguishing between single object action recognition and interactive group activity recognition.

A method for modeling an interactive activity according to an exemplary embodiment includes receiving a data set of a two-dimensional skeleton position extracted from a video, calculating position coordinates of the object from the input data set, Comprising the steps of: calculating tracking characteristics including a motion velocity and a motion direction of an object; determining a mutual object based on the mutual zone corresponding to the object and the calculated tracking characteristics; determining a skeleton data Calculating features from the set, and modeling the calculated features into topics of single human actions and intergroup activities.

The computing the positional coordinates of the object according to an embodiment includes detecting positional coordinates for the object using four joints of the torso from the input data set.

The mutual objects and the non-mutual objects corresponding to the mutual object according to an embodiment are determined through the mutual potential zone and the tracking characteristics.

The step of calculating tracking characteristics including the motion velocity and the motion direction of the object according to an embodiment may include extracting the motion direction between the spatially-temporal joint distance of the object and the human objects, .

The modeling step according to an embodiment includes generating a probability model for the single human actions and the mutual group activities using a modeling algorithm.

According to an embodiment of the present invention, there is provided a method for identifying mutual objects, comprising the steps of receiving position coordinates of an object, determining single mutual potential zones located within a predetermined range from the object based on position coordinates of the input object, Calculating a ratio of overlapping regions for each object based on single mutual potential zones; and comparing the calculated ratio with a threshold assigned to a group ID for each object to identify the object.

The step of determining the single mutual potential zones according to an embodiment includes determining the single mutual potential zones based on the position coordinates of the object and the radius of the circle.

The ratio of the single mutual potential zones and the overlap region according to one embodiment is identified for each object.

The ratio according to one embodiment is compared to a threshold value for determining the group ID of each object.

및

중에서 적어도 하나 이상의 좌표에 대한 특징들을 추출하는 단계, 상기 추출된 특징들에 상응하는 데이터세트를 인식하는 단계, 및 상기 인식된 데이터세트에 기초하여 인트라 오브젝트 데이터 세트와 인터 오브젝트 특징 데이터세트를 획득하는 단계를 포함한다.A method of configuring a feature data set according to an exemplary embodiment includes receiving a group ID, comparing the number of objects for each group corresponding to the group ID,

And

Extracting features for at least one of the plurality of coordinates from the extracted data set, recognizing a data set corresponding to the extracted features, and obtaining an intra-object data set and an inter-object characteristic data set based on the recognized data set .

The step of extracting features in accordance with an embodiment includes extracting features for x = y coordinates when there is one object in the group.

및

의 좌표에 대한 특징들을 추출하는 단계를 포함한다.The step of extracting the features according to an embodiment may comprise: if there is more than one object in the group,

And

And extracting features for the coordinates of the coordinates.

The method of constructing a feature data set according to an embodiment may further include classifying the object into two groups of non-mutual objects and mutual objects in consideration of the comparison result of the number of objects.

The step of classifying the objects into two groups of non-mutual objects and mutual objects according to an exemplary embodiment includes extracting spatial-temporal joint distances and motion direction characteristics for the objects.

The step of extracting the features according to an embodiment includes extracting at least one or more features from the data set and the inter-object characteristic for the intra-object characteristic and the mutual activity recognition for single action recognition.

A method for generating a probability model according to an exemplary embodiment includes receiving a feature data set, clustering features in the feature data sets into codewords by applying a K-average clustering algorithm, Mapping inter-object features to codeword histograms of actions and activities, encoding words using a hierarchical model based on the mapped histogram, and using the encoded words to calculate a probability And outputting the model.

The step of outputting the probability model according to an embodiment includes generating the probability model based on topic modeling based on the metrology model.

The interactive modeling program according to an embodiment includes an instruction set that receives a data set of a two-dimensional skeleton position extracted from a video, an instruction set that calculates position coordinates of the object from the input data set, A set of instructions for determining a mutual object based on the calculated tracking characteristics and a mutual zone corresponding to the object, a set of instructions for determining a mutual object corresponding to the determined mutual object And a set of instructions for modeling the computed features into topics of single human actions and mutual group activities.

According to embodiments, human interactive actions can be represented from video captured from surveillance cameras.

According to embodiments, interactions between human objects can be modeled within a group or between groups.

According to embodiments, omitting non-interacted objects may reduce the computational cost.

According to embodiments, the quality of the training data set of the function can be improved since the singular value is not included in the data set.

According to embodiments, the classification accuracy can be improved in distinguishing between single object action recognition and interactive group activity recognition.

1 is a flowchart of a modeling method for single action and mutual activity recognition.
Fig. 2 is a diagram showing the 14-joint human posture, the determination of the center point, the distance of the object and the direction of movement.
FIG. 3 is a view for explaining interaction zone determination and object group establishment. FIG.
4 is a diagram for explaining the determination of four features using joint position information.
Figure 5 shows the process of interaction zone identification and object group creation.
FIG. 6 shows a process of configuring two feature data sets divided into an intra-object feature data set and an inter-feature feature data set.
Figure 7 shows the process of codebook generation and topic modeling for two feature data sets.
Fig. 8 is a diagram for explaining an embodiment for mapping one feature vector to a histogram of code words. Fig.
9 is a diagram showing a hierarchical model for a topic model of a four-level structure.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, the scope of the rights is not limited or limited by these embodiments. Like reference symbols in the drawings denote like elements.

The terms used in the following description are chosen to be generic and universal in the art to which they are related, but other terms may exist depending on the development and / or change in technology, customs, preferences of the technician, and the like. Accordingly, the terminology used in the following description should not be construed as limiting the technical thought, but should be understood in the exemplary language used to describe the embodiments.

Also, in certain cases, there may be a term chosen arbitrarily by the applicant, in which case the meaning of the detailed description in the corresponding description section. Therefore, the term used in the following description should be understood based on the meaning of the term, not the name of a simple term, and the contents throughout the specification.

1 is a flowchart of a modeling method for single action and mutual activity recognition.

In an interactive modeling method according to an exemplary embodiment, a data set of a two-dimensional skeleton location extracted from a video is input (step 101).

The input data includes a human object skeleton with joint position information.

The output corresponding to the input data is a probability model based on the single object actions and interaction group activities needed to support the classifier. To this end, a method for modeling an interaction according to an embodiment calculates the positional coordinates of an object from an input data set (step 102). That is, in order to estimate the position of the joint, it is necessary to determine the position of the joint. For this purpose, the position coordinates of the object are calculated through the step 102.

For example, the interactions modeling method may detect positional coordinates for an object using four joints of the torso from the input data set to calculate the positional coordinates of the object.

According to Yang et al., Each human posture includes 14 joint elements.

14 joint elements will be described later in detail with reference to FIG.

Next, a method for modeling an interaction according to an exemplary embodiment calculates tracking characteristics including the movement speed and the direction of movement of the object from the coordinates of the calculated object (step 103). In addition, the method for modeling an interaction according to an exemplary embodiment may further include determining a mutual object based on the mutual zone corresponding to the object and the calculated tracking characteristics (step 104), extracting features from the data set of the skeleton for the determined mutual object (Step 105).

For example, in order to calculate tracking characteristics, an interactive modeling method according to one embodiment extracts the spatial-temporal joint distance of an object and the direction of motion between human objects, and extracts from the location data set of the skeleton.

The method of interacting modeling according to an exemplary embodiment models the calculated features to topics of single human actions and mutual group activities (step 106), and generates a probability model using the modeling result (step 107).

For example, the interactions modeling method can generate a probability model and intergroup activities for single human actions using a modeling algorithm to model intergroup activities.

Fig. 2 is a diagram showing the 14-joint human posture, the determination of the center point, the distance of the object and the direction of movement.

As shown in FIG. 2, the human posture 201 includes 14 joint elements. In addition, to localize the coordinates of an object, the center point must be determined using four joints of the body 202. [

Specifically, each point shown in Fig. 2 can be calculated by the following equation (1) as the coordinates of the object.

[Equation 1]

는

번째 프레임에 나타나는 휴먼 오브젝트(x)의 몸통에 대한 조인트 위치 좌표들이다.here,

The

Th frame of the human object (x).

와 방향

을 가정한다.The tracking algorithm is based on the motion velocity < RTI ID = 0.0 >

And direction

.

와

프레임에 상응하는

와

로부터의 트래킹 특징들은 아래 [수학식 2] 및 [수학식 3]의 오브젝트 좌표들로부터 계산될 수 있다.

Wow

Frame-equivalent

Wow

Can be calculated from the object coordinates of [Equation 2] and [Equation 3] below.

&Quot; (2) "

&Quot; (3) "

What is important to the interaction representation is how to identify objects that interact with other objects in the current scene.

The following advantages can be obtained by performing the identification process before proceeding with the feature extraction step instead of calculating the features extracted from all the detected objects.

By omitting non-interacted objects, the computational cost can be reduced.

Since the singular values are not included in the data set, the quality of the training data set of the function can be improved. This means that single objects are not considered for detection and recognition according to the interaction.

Classification accuracy can be improved in distinguishing between single object action recognition and interactive group activity recognition.

In the present invention, an IPZ (Interaction Potential Zone) algorithm can be used.

The IPZ (Interaction Potential Zone) algorithm will be described in detail with reference to FIG.

FIG. 3 is a view for explaining interaction zone determination and object group establishment. FIG.

As shown in FIG. 3, the IPZ (Interaction Potential Zone) algorithm is a basic unit necessary for detecting the Group Interaction Zone (GIZ)

을 갖는 서클로 정의된다.Each object has an operating zone. The operating zone is located in the periphery of the object,

≪ / RTI >

로 식별된다.Thus, based on the object center coordinates 501,

≪ / RTI >

Next, the ratio 503 of overlapping regions between IPZs calculated as 301 in Fig. 3 is calculated. The ratio 503 at this time is the ratio of the overlapping area to the entire area covered by the interactive human objects, and can be calculated by Equation (4).

&Quot; (4) "

는

휴먼 오브젝트의 IPZ이다.here,

The

The IPZ of the human object.

는 중첩되는 IPZ들을 갖는 사람 수이다.

Is the number of people with overlapping IPZs.

이면, 도 3의 도면부호 301과 같이 홀로 서있는 단 하나의 오브젝트가 있는 경우이고, 이 때의

은 이러한 파라미터의 결과이다.if,

, There is a case where there is only one object alone, as indicated by reference numeral 301 in Fig. 3. In this case,

Is the result of these parameters.

The set of human objects has interactions that are assigned through a comparison operation that can be performed by Equation (5).

&Quot; (5) "

는 휴먼 오브젝트들의 세트가 같은 그룹에 놓일 가능성이 얼마나 되는지의 경향을 제어하는 임계 값이다.here

Is a threshold that controls the likelihood that a set of human objects will be placed in the same group.

Group assignment can be explained in three situations.

에 상응하여, 중복 영역 없이 홀로 서있는 경우라면, 새로운 그룹 식별자(GID, GroupID)가 할당된다.If the current object is

, A new group identifier (GID, GroupID) is assigned if it stands alone without overlapping areas.

에 따라 작은 면적에서 중첩되는 경우, 이 두 오브젝트들은 새로운 다른 그룹 식별자(GID, GroupID)가 할당된다.If the object is different from the current object

, The two objects are assigned a new different group identifier (GID, GroupID).

의 상태를 만족하는 경우, 이 두 오브젝트는 같은 그룹 식별자(GID, GroupID)가 할당된다.If the two objects are currently overlapping

, The two objects are assigned the same group identifier (GID, GroupID).

The output is a set for the object's group identifier (GID, GroupID).

However, there are special cases for assigning group identifiers (GIDs, GroupIDs).

For example, as shown at reference numeral 301 in FIG. 3, an object of GID = A indicates a tendency to move to a group in the object group direction of GID = B.

For this situation, it is necessary to assign a group identifier (GID, GroupID).

In this situation, consideration of dynamic objects, ie, F_v ^ X (T-1, T) ≥δ, must be taken into account in that the speed threshold is shifted or non-moving to identify the δ and the current entity.

In the speed and direction of the movement from the current position, the position of the object is calculated at the next time (next time (t + 1)) as shown in Equation (6).

&Quot; (6) "

는 [수학식 7]과 같이

로 식별되는 다음 목적지 그룹(next destination group)의 그룹 식별자(GID, GroupID)로 변경된다.If the next position value is within the IPZ of another group, the group identifier (GID, GroupID) of the object,

Is expressed by Equation (7)

To the group identifier (GID, GroupID) of the next destination group identified by the destination identifier.

&Quot; (7) "

The case shown at reference numeral 302 is as shown in FIG.

로 식별되는 오브젝트의 그룹 식별자는 [수학식 8]과 같이 새로운 그룹 식별자로 변경된다. 이때의 새로운 그룹 식별자는 이미 존재하는 GID들(

로 표시)과 중첩되지 않는다.If the next position value is outside the IPZ of another group,

Is changed to a new group identifier as in Equation (8). At this time, the new group identifier includes already existing GIDs (

(Not shown).

&Quot; (8) "

May be applied to the situation indicated at 302 in FIG.

To illustrate the relationship between object joints for single object action recognition and the relationship between two objects' joints for group activity recognition interacting, the present invention proposes a joint-configuration in a space-time dimension And extracts the distance and direction between them.

4 is a diagram for explaining the determination of four features using joint position information.

Specifically, the space-time joint features are calculated based on the skeleton location as shown at 401-404.

Spatial joint distance 401 can be defined as a pairwise Euclidean distance for all joints between two people in a frame. That is, the spatial joint distance 401 captures the interaction attitude and the distance between two joints using Equation (9).

&Quot; (9) "

과

는

에 상응하는 시간

에서의 휴먼 오브젝트

와

의 조인트 i와 j에 대한 2D 위치 좌표이다.here,

and

The

Corresponding time

Human object in

Wow

Lt; / RTI > is the 2D position coordinate for the joints i and j of FIG.

인 경우나 사람들 간

에 측정될 수 있다.This is one person (

Or between people

Lt; / RTI >

과

의 프레임에 상응하는 time

와

에서의 상호 작용의 사지 쌍 간 거리를 측정한다.Temporal joint distance is defined as the Euclidean distance between all pairs of two joints in different frames. That is, the temporal joint distance is calculated based on (10)

and

Corresponding to the frame of time

Wow

The distance between the pairs of limbs is measured.

&Quot; (10) "

또는 두 사람 간

에 측정될 수 있다.This is a person

Or between two people

Lt; / RTI >

Spatial joint motion 403 captures the angle between two joints in the interactive posture, which can be calculated based on (11).

&Quot; (11) "

또는 두 사람 간

에 측정될 수 있다.This is a person

Or between two people

Lt; / RTI >

와

에 해당하는 프레임의

과

에서 측정될 수 있다. 예를 들면, [수학식 12]를 통해 상호 작용의 수족 쌍에 대한 각도를 측정할 수 있다.In the case of time joint motion 404, it can be defined between every pair of joints,

Wow

Of the frame corresponding to

and

Lt; / RTI > For example, it is possible to measure the angle with respect to the lumbar pair of the interaction through [Equation 12].

&Quot; (12) "

또는 두 사람 사이인 경우

에서 모두 측정될 수 있다.The angle for the pair of interactions is one person

Or between two people

Lt; / RTI >

Figure 5 shows the process of interaction zone identification and object group creation.

FIG. 5 collects the object center coordinates (step 501). In order to identify a mutual object, coordinates of an object are input.

Next, in order to generate the interaction zone identification and the object group, a single object area can be established (step 502) and the overlap ratio can be calculated (step 503).

It is possible to determine single mutual potential zones located within a predetermined range from the object based on the position coordinates of the input object and calculate the ratio of the overlap region for each object based on the determined single mutual potential zones. For example, to determine single mutual potential zones, single mutual potential zones can be determined based on the object's location coordinates and the radius of the circle.

In addition, the object can be identified by comparing the calculated ratio with the threshold assigned to the group ID for the object. In one example, the ratio of the single mutual potential zones to the overlap region can be identified for each object, and the ratio at this time can be compared to a threshold value for determining the group ID of each object.

Next, in order to generate the interaction zone identification and the object group, the radius may be considered in step 504 (step 504).

을 갖는 서클로 정의된다.Each object has an operating zone. The operating zone is located in the periphery of the object,

≪ / RTI >

로 식별될 수 있다.Thus, based on the object center coordinates,

Lt; / RTI >

The ratio 503 of the overlapping regions among the calculated IPZs is the ratio of the overlapping regions to the entire region covered by the interactive human objects.

에 상응하여, 중복 영역 없이 홀로 서있는 경우라면, 새로운 그룹 식별자(GID, GroupID)가 할당된다(단계 506).If the current object is

, A new group identifier (GID, GroupID) is assigned (step 506).

에 따라 작은 면적에서 중첩되는 경우, 이 두 오브젝트들은 새로운 다른 그룹 식별자(GID, GroupID)가 할당된다(단계 505).If the object is different from the current object

, The two objects are assigned a different group identifier (GID, GroupID) (step 505).

의 상태를 만족하는 경우, 단계 506을 통해 같은 그룹 식별자(GID, GroupID)가 할당된다.If the two objects are currently overlapping

The same group identifier (GID, GroupID) is allocated in step 506. [

The output is a set for the object's group identifier (GID, GroupID) (step 507).

However, there are special cases for assigning group identifiers (GIDs, GroupIDs).

For example, as shown at reference numeral 301 in FIG. 3, an object of GID = A indicates a tendency to move to a group in the object group direction of GIDB.

For this situation, it is necessary to assign a group identifier (GID, GroupID).

FIG. 6 shows a process of configuring two feature data sets divided into an intra-object feature data set and an inter-feature feature data set.

In FIG. 6, the feature extraction process in step 105 of FIG. 1 will be described in detail.

A method of configuring a feature data set according to an exemplary embodiment may first receive a group ID (step 601).

That is, in order to extract the feature data set, the objects must be distinguished into two groups based on the group ID. For example, based on the group ID, objects should be classified into two groups, non-cross-object and cross-object, and based on this, the spatial-temporal joint distance and motion direction characteristics for the object should be extracted. To this end, at least one or more features of an intra-object feature for single-action recognition and a data set and an inter-object feature for mutual activity recognition are extracted.

Next, in the method of constructing a feature data set according to an embodiment, it is possible to determine whether the number of objects is equal to or greater than 2 by comparing the number of objects for each group corresponding to the group ID (step 602).

및

중에서 적어도 하나 이상의 좌표에 대한 특징들을 추출할 수 있다.A method of constructing a feature data set according to an embodiment includes:

And

The features of at least one of the coordinates can be extracted.

For this purpose, when the number of objects is two or more as a result of the determination in step 602, the method of constructing the feature data set extracts features for x = y coordinates when there is one object in the group (step 603).

및

의 좌표에 대한 특징들을 추출할 수 있다(단계 604).If the number of objects is not equal to or greater than 2 as a result of the determination in step 602, if there is more than one object in the group,

And

(Step 604). ≪ RTI ID = 0.0 >

A method of constructing a feature data set according to an embodiment may recognize a data set corresponding to the extracted features (step 605).

Features can be used to recognize single actions without testing interaction with other objects.

와

의 컨디션에서 계산될 수 있다.If the current object has the same group ID as another object, it means that the groups are made up of more objects. These features may be implemented as in steps 603 and 604

Wow

Can be calculated.

의 조건에서 계산되어야 한다.Specifically, if only one object is in the group,

Should be calculated.

In addition, an intra-object feature data set (step 606) and an inter-object feature data set may be acquired (step 607) based on the recognized data set.

The extracted features may include a spatial joint distance feature subset, a temporal joint distance feature subset, a spatial joint motion feature subset, a temporal joint motion feature subset, And a temporal joint motion feature subset.

으로 표현될 수 있다.First, the spatial joint distance feature subset

. ≪ / RTI >

으로 표현될 수 있고, 공간 조인트 모션 특징 서브셋(Spatial joint motion feature subset)은

으로 표현될 수 있다.The Temporal joint distance feature subset

And the spatial joint motion feature subset may be expressed as

. ≪ / RTI >

으로 표현될 수 있다.The Temporal joint motion feature subset

. ≪ / RTI >

A vector for expressing a feature extracted from a single human object can be expressed as follows.

Further, the inter-object characteristic data set may include the following components.

, 임시 조인트 거리(Temporal joint distance)으로서

, 공간 조인트 모션(Spatial joint motion)으로서

, 임시 조인트 모션(Temporal joint motion)으로서

를 포함한다.As a spatial joint distance,

, As a temporary joint distance

, As a spatial joint motion

, As a temporary joint motion

.

In order to express a feature extracted from an interactive human object, a vector feature can be expressed as follows.

The two feature data sets are collected frame by frame according to the input video and the two-dimensional mattress. One of the two feature data sets is for an intra-object feature data set and the other is for an inter-object feature data set.

Figure 7 shows the process of codebook generation and topic modeling for two feature data sets.

7, the modeling process of step 106 of FIG. 1 will be described in detail.

A dual structure model including an intra-object feature data set 701 and an inter-object feature data set 702 can be used to generate a probability model.

The model is developed based on the assumption of the "bag-of-words" approach, ie, the pachinko paring model.

The statistical analysis can be analyzed based on the histogram due to the concurrent occurrence of words, and in order to support this model, the codebook can be subjected to K-means clustering as in step 703. That is, the probability model generation method may perform clustering of the features in the feature data sets into codewords by applying a K-average clustering algorithm (step 703).

Next, the probability model generation method may map intra-object features 704 and inter-object features 705 according to clustering to codeword histograms of actions and activities.

Next, the probability model generation method encodes the words using a hierarchical model based on the mapped histogram (step 706).

Then, the probability model generation method outputs the probability model using the encoded words. Specifically, a probability model for an action may be output (step 707) or a probability model for activity may be output (step 708).

The probability model generation method can generate a probability model based on topic modeling based on a metric model to output a probability model.

Fig. 8 is a diagram for explaining an embodiment for mapping one feature vector to a histogram of code words. Fig.

This corresponds to an embodiment in which an intra-object vector is mapped to a histogram.

에 해당하는 인트라 오브젝트 벡터를 워드들의 수로 표현되는 히스토그램에 맵핑할 수 있다.In other words,

Can be mapped to a histogram represented by the number of words.

9 is a diagram showing a hierarchical model for a topic model of a four-level structure.

To learn and recognize based on the "bag-of-words" model, it should be developed from a flexible and expressive Latent Dirichlet Allocation (LDA) such as the Pachinko Allocation Model (Li et al.

9 consists of N action words or M interaction words at the bottom level, consisting of n ₁ action subtopics at the first level, m ₁ interaction subtopics at the bottom level and n ₂ action subtopics, and m ₂ interaction subtopics, and one level at the top level.

A full report on this model is given in Li (2006).

As a result, the present invention can be used to reduce the calculation cost by omitting non-interacted objects. In addition, because no singular values are included in the data set, it is possible to improve the quality of the training data set of the function and to distinguish between single object action recognition and interactive group activity recognition So that the accuracy of the classification can be improved.

The method according to an embodiment of the present invention can be implemented in the form of a program command which can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions recorded on the medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

While the invention has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. This is possible.

Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined by the equivalents of the claims, as well as the claims.

Claims (19)

A method of interactions modeling, at least temporarily implemented by a computer,
Receiving a data set of a two-dimensional skeleton position extracted from a video;
Computing positional coordinates of an object from the input data set;
Calculating tracking characteristics including a motion velocity and a motion direction of the object from coordinates of the calculated object;
Determining a mutual object based on the mutual zone corresponding to the object and the calculated tracking characteristics;
Computing features from a data set of skeletons for the determined cross-object; And
Modeling the computed features into topics of single human actions and mutual group activities
/ RTI >
The method according to claim 1,
Wherein calculating the position coordinates of the object comprises:
Detecting position coordinates for the object using four joints of the torso from the input data set
/ RTI >
The method according to claim 1,
Wherein the non-mutual objects corresponding to the mutual object and the mutual object are determined through the mutual potential zone and the tracking characteristics.
The method according to claim 1,
Wherein the step of calculating tracking characteristics including the motion velocity and the motion direction of the object comprises:
Extracting the motion direction between the spatially-temporal joint distance of the object and the human objects, and extracting from the position data set of the skeleton
/ RTI >
The method according to claim 1,
Wherein the modeling comprises:
Generating a probability model for the single human actions and the mutual group activities using a modeling algorithm;
/ RTI >
A method of inter-object identification at least temporarily implemented by a computer,
Receiving position coordinates of an object;
Determining single mutual potential zones located within a predetermined range from the object based on position coordinates of the input object;
Calculating a ratio of overlapping regions for each object based on the determined single mutual potential zones; And
Identifying the object by comparing the calculated ratio with a threshold assigned to the group ID for each object
/ RTI >
The method according to claim 6,
Wherein determining the single mutual potential zones comprises:
Determining the single mutual potential zones based on the position coordinates of the object and the radius of the circle
/ RTI >
The method according to claim 6,
Wherein the ratio of the single mutual potential zones to the overlap region is identified for each object.
The method according to claim 6,
Wherein the ratio is in contrast to a threshold value for determining a group ID of each object.
A method of constructing a feature data set that is at least temporarily implemented by a computer,
Receiving a group ID;
Comparing the number of objects for each group corresponding to the group ID;
Considering the number of the compared objects

And

Extracting features for at least one of the coordinates;
Recognizing a data set corresponding to the extracted features; And
Acquiring an intra-object data set and an inter-object characteristic data set based on the recognized data set
Gt; a < / RTI >
11. The method of claim 10,
Wherein extracting the features comprises:
If there is one object in the group, extracting features for x = y coordinates
Gt; a < / RTI >
11. The method of claim 10,
Wherein extracting the features comprises:
If there is more than one object in the group,

And

Extracting features for the coordinates of the coordinates
Gt; a < / RTI >
11. The method of claim 10,
Classifying the object into two groups of non-mutual objects and mutual objects in consideration of the comparison result of the number of objects
Further comprising the steps of:
14. The method of claim 13,
The step of classifying the objects into two groups of non-
The spatial-temporal joint distance and motion direction features for the object are extracted
Gt; a < / RTI >
11. The method of claim 10,
Wherein extracting the features comprises:
Extracting at least one characteristic from among a data set and an inter-object characteristic for recognizing an intra-object characteristic and a mutual activity for single action recognition
Gt; a < / RTI >
A method of generating a probability model at least temporarily implemented by a computer,
Receiving a feature data set;
Clustering features in the feature data sets into codewords by applying a K-means clustering algorithm;
Mapping intra-object features and inter-object features according to the clustering to codeword histograms of actions and activities;
Encoding the words using a hierarchical model based on the mapped histogram; And
Outputting a probability model using the encoded words
A probability model generating step of generating a probability model;
17. The method of claim 16,
Wherein the outputting of the probability model comprises:
Generating the probability model based on topic modeling based on the metrology model
A probability model generating step of generating a probability model;
A computer-readable recording medium having recorded thereon a program for carrying out the method according to any one of claims 1 to 17.
17. An interaction modeling program stored on a recording medium, the program being run on a computing system,
A command set that receives a data set of a two-dimensional skeleton position extracted from a video;
A set of instructions for computing positional coordinates of an object from the input data set;
Calculating a tracking feature including a motion velocity and a motion direction of the object from the position coordinates of the calculated object;
A set of instructions for determining a mutual object based on the mutual zone corresponding to the object and the calculated tracking characteristics;
A set of instructions for computing features from a data set of skeletons for the determined cross-object; And
A set of instructions for modeling the computed features into topics of single human actions and mutual group activities
A program of interactive activity modeling comprising:

Images