JP2014026455A - Media data analysis device, method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a media data analysis device capable of giving a piece of precise tag information even under a situation where sufficient scale of learning data is hardly obtained.SOLUTION: A dimension reduced space learning section 3 learns a matrix representing a map for reducing the number of dimensions of a feature vector by applying a dimension reduction technique with semi-teacher to generate feature vectors whose dimensions have been reduced. A classifier learning section 4 learns a classifier for each tag based on a posterior probability estimation based on the feature vectors whose dimensions have been reduced. A feature dimension reduction section 6 generates a feature vector whose dimension has been reduced from extracted feature vector with respect to input media data. A classifier estimation section 7 classifies whether the input media data should be given with a tag on each tag based on the learnt classifier and outputs a tag vector.

Description

  The present invention relates to a media data analysis apparatus, method, and program.

  Automatic media that acquires the components (scenes) included in the media such as images and music, scenes, and actions, and consequently the semantics that are formed by their mixed connection, and presents them to the user in the form of text labels via linguistic information Although the annotation problem (FIG. 11 (a)) is one of the important issues from the early days in the pattern recognition field, it has not yet reached an essential solution.

  Further, as a problem similar to the automatic annotation problem, there is a media search problem (FIG. 11B) in which semantics envisioned by the user are given to the computer in the form of text and an image matching the text is presented. Considering the correspondence between semantics, language information, and media, this problem can be regarded as a dual problem of automatic media annotation and, like automatic media annotation, is an important issue in the field of pattern recognition. However, this problem has not yet been resolved. In recent years, media annotation retrieval has been actively studied to describe these two dual problems in a unified framework. The present invention relates to the problem of annotation on media (especially acoustic signals) based on a machine learning approach.

  One of the most important issues in the machine learning approach is how many good quality learning data can be collected. Although it is becoming possible to acquire a large amount of labeled media through media sharing sites such as Flickr and Last.fm, which have been developing significantly in recent years, the reliability of labels collected from these sites is not necessarily high. On the other hand, collecting or creating a large amount of media with highly reliable labels is labor intensive. From these discussions, a machine learning method using a small number of media with highly reliable labels and a large amount of media with low label reliability plays an important role. In particular, we focus on semi-supervised learning that discards unreliable labels and treats them as unlabeled data.

  Traditionally, designing a latent variable space that takes into account the co-occurrence relationship between media and labels, and learning the nature that semi-supervised learning can be applied by making the adjacency relationship in the latent variable space strongly reflect the similarity of labels Studies given to data are known (for example, Non-Patent Document 1, Non-Patent Document 2).

Shogo Kimura, Masaru Sugiyama, Takuho Nakano, Hirokazu Kameoka, Akira Sakano, “SSCDE: Semi-Teacher Canonical Density Estimation Method for Image Recognition Retrieval,” Proceedings of Symposium on Image Recognition and Understanding, 2010. J. Takagi, Y. Ohishi, A. Kimura, M. Sugiyama, M. Yamada, and H. Kameoka, “Automatic audio tag classification via semi-supervised canonical density estimation,” in Proc. ICASSP, pp. 2232-2235, 2011.

  In the conventional research shown in Non-Patent Documents 1 and 2, the classifier used for annotation is designed on a case basis, so sufficient annotation accuracy cannot be obtained unless very large learning data is prepared. There is a problem.

  The present invention has been made to solve the above problems. An object of the present invention is to provide a media data analysis apparatus, method, and program capable of providing tag information with high accuracy even in a situation where learning data of a sufficient scale cannot be obtained.

  In order to achieve the above object, a media data analysis apparatus according to the present invention is a media data analysis apparatus that adds tag information, which is information for explaining the content data, to given content data, Content data that is an element of a tagged learning data set that is a set of content data to which tag information is assigned in advance, and content data that is an element of an untagged learning data set that is a set of content data to which tag information is not given A learning data feature extracting means for extracting a feature vector, which is a vector expressing the characteristics of the content data, and the feature vector extracted for each content data in the tagged learning data set and the pre-assigned Tag information and each content of the untagged learning data set Based on the feature vector extracted for data, a dimension reduction space learning means for learning a matrix representing a mapping for reducing the number of dimensions of the feature vector, and the dimension reduction space learning means learned by the dimension reduction space learning means Dimension-reduced feature generating means for generating the feature vector with reduced dimensions from each of the feature vectors extracted for each content data of the tagged learning data set based on a matrix; and the tagged learning data A posteriori for classifying tag information to be assigned to content data based on the feature vector with reduced dimensions and the pre-assigned tag information generated by the dimension reduction feature generation means for each piece of content data in the set Classification model learning means for learning a classification model based on probability estimation and an input controller From the feature vector extracted for the input content data based on the matrix learned by the dimension reduction space learning means and the input data feature extraction means for extracting the feature vector from the component data Feature dimension reduction means for generating the reduced feature vector, the feature vector generated by the feature dimension reduction means, and the input content data based on the classification model learned by the classification model learning means And classifying means for classifying the tag information to be assigned to.

  A media data analysis method according to the present invention is a media data analysis method in a media data analysis apparatus for adding tag information, which is information for explaining content data, to given content data. Content data that is an element of a tagged learning data set that is a set of content data to which the tag information is assigned in advance by the extraction means, and an untagged learning data set that is a set of content data to which no tag information is given. The feature vector, which is a vector representing the characteristics of the content data, is extracted from each of the content data as elements, and the feature vector extracted for each content data of the tagged learning data set by the dimension reduction space learning means. And pre-assigned tag information and A matrix representing a mapping for reducing the number of dimensions of the feature vector is learned based on the feature vector extracted for each content data of the untagged learning data set, Based on the matrix learned by the dimension reduction space learning means, each of the feature vectors extracted for each content data of the tagged learning data set is generated, and the feature vectors with reduced dimensions are generated and classified. Based on the feature vector with reduced dimensions and the pre-assigned tag information generated by the dimension reduction feature generation unit for each content data in the tagged learning data set by the model learning unit, Classification model based on posterior probability estimation for classifying tag information to be assigned The feature vector is extracted from the input content data by the input data feature extraction means, and the input is performed based on the matrix learned by the dimension reduction space learning means by the feature dimension reduction means. The feature vector with reduced dimensions is generated from the feature vector extracted with respect to the extracted content data, and the feature vector generated by the feature dimension reduction unit and the classification model learning unit learn by the classification unit Based on the classification model, the tag information to be given to the input content data is classified.

  The program according to the present invention is a program for assigning tag information, which is information for explaining the content data, to the given content data, the computer providing the content data to which the tag information is assigned in advance. Characteristics of content data from each of content data that is an element of a tagged learning data set that is a set of and content data that is an element of an untagged learning data set that is a set of content data to which no tag information is given. Learning data feature extracting means for extracting a feature vector that is a vector to be expressed, the feature vector extracted for each content data of the tagged learning data set, the tag information given in advance, and the untagged learning data Extracted for each content data of the set A dimension reduction space learning means for learning a matrix representing a mapping for reducing the number of dimensions of the feature vector based on the vector, and the tag based on the matrix learned by the dimension reduction space learning means Dimension reduction feature generation means for generating the feature vector with reduced dimensions from each of the feature vectors extracted for each content data in the tagged learning data set, and the dimension for each content data in the tagged learning data set Learning a classification model based on posterior probability estimation for classifying tag information to be assigned to content data based on the feature vector with reduced dimensions generated by the reduction feature generation means and the tag information given in advance. The classification model learning means for performing the feature vector from the input content data Based on the matrix learned by the input data feature extraction means and the dimension reduction space learning means, the feature vector with reduced dimensions is generated from the feature vector extracted for the input content data. Classifying tag information to be given to the input content data based on the feature dimension reducing means, the feature vector generated by the feature dimension reducing means, and the classification model learned by the classification model learning means It is a program for functioning as a classification means.

  As described above, according to the media data analysis apparatus, method, and program of the present invention, the feature vector is based on the feature vector extracted for each content data of the tagged learning data set and the untagged learning data set. Learning a matrix that represents a mapping to reduce the number of dimensions, and using a feature vector with reduced dimensions for each content data feature vector in a tagged learning data set, a classification model based on posterior probability estimation Even in situations where sufficient scaled learning data cannot be obtained by classifying tag information based on the learned classification model for feature vectors that have been learned and reduced in dimension with respect to feature vectors of input content data. The effect that tag information can be provided with high accuracy is obtained.

(A) It is a figure for demonstrating preparation of a tag vector, (b) It is a figure for demonstrating learning of a classifier. It is a diagram for explaining a method of calculating; | (α x a) function value q _i. It is a figure for demonstrating the multiplication method of basis function vector (phi) (x, 0). It is a figure for demonstrating the learning method of the classifier within the framework of semi-teacher learning. It is a block diagram which shows the example of 1 structure of the media data analyzer which concerns on the 1st Embodiment of this invention. It is a figure for demonstrating the method of extracting a feature vector. It is a flowchart which shows the content of the classifier learning process routine in the media data analyzer which concerns on the 1st Embodiment of this invention. It is a flowchart which shows the content of the tag provision process routine in the media data analyzer which concerns on the 1st Embodiment of this invention. It is a block diagram which shows the example of 1 structure of the media data analyzer which concerns on the 2nd Embodiment of this invention. It is a figure which shows a ROC curve and AUC. (A) It is a figure for demonstrating a media annotation, (b) It is a figure for demonstrating a media search.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  First, the principle of adding a tag vector to content data will be described.

<Overview>
By designing supervised and semi-supervised classifiers using an approach based on discriminative learning, we take an approach that can ensure high accuracy even in situations where learning data of sufficient scale cannot be obtained. In particular, using support vector machines (SVM), least squares probabilistic classifiers (LSPC), and Laplacian SVM as classifiers, this classifier is driven on the latent variable space considering the co-occurrence relationship between media and labels. By doing so, it enables highly accurate annotation.

<Automatic tagging method based on supervised learning>
<Formulation of problem>
For example, when an audio signal is considered as media data, automatic tagging of media data is performed when a sound material s _{q serving as} a query is input, and a set of tag types V = {v ₁ , v ₂ , ..., v _{| V |} }, and selecting and assigning an appropriate tag that describes the sound material s _q . Here, s _q is an acoustic signal such as a sound source or sound effect recorded by field recording, and each tag v _i is included in the sound material such as “bird's voice”, “rain”, “car”, etc. For example, a word representing an entire sound material such as sound or "forest" or "town", or a word representing an application such as "loop music" or "sound effect" if it is a material or sound effect for loop music. The output for the input s _q is represented by a vector y _q having a number of dimensions equal to the number of tag types | V |. That is, the value y _{q, i} of the i-th element of the vector y _q is set to 1 when the i-th tag v _i is _added, and 0 when not added. Hereinafter, a vector representing such tag information is referred to as a tag vector. When the feature vector extracted from the media data (sound material in the above description) s _q is represented by x _q , the above problem is a function f ′ (•) that takes x _q as an input and outputs an appropriate tag vector y _q Can be expressed as a problem.

f (・) is called a decision function, and for the input vector x _q , a score that is a criterion for deciding whether or not each tag should be assigned is calculated, and this is the number of dimensions equal to the number of tag types | V | Output as a real vector with. Also, g (•) is called a discriminant function, and outputs a binary vector y _q having a number of dimensions equal to the number of tag types | V | from the real vector obtained by the decision function f (•). . In general, it is complicated to obtain a function that performs such multi-dimensional output at once. Therefore, the present invention simplifies the problem by assuming that the dimensions of this output are independent of each other. That is, as shown in FIG. 1, the determination of whether or not to grant the tag performs for each tag together all of them, to create a tag vector y _q. Accordingly, the function f ′ (•) can be expressed as follows using | V | functions f ′ _i (•) that determine the output for each tag.

However, ^{X T} represents the transpose of vector or matrix X.

In other words, this problem is divided into a set of | V | problems that find a function f ′ _i (•) that outputs binary values. Since each problem is a problem for determining whether or not to attach a tag v _i when an input x _q is given, it can be solved in the framework of a binary classification problem. That is, it is only necessary to solve the problem of classifying x _q into one of the class to which the tag v _i is assigned and the class to which the tag v _i is not given. In the present invention, this binary classification problem is solved using a classification method based on machine learning.

As an example of a classification method based on machine learning, there is a method in which a certain score as a reference for classification is first calculated for the input x _q , and classification is performed by comparing the score with a predetermined threshold value. . That is, when the threshold for the function f _i (•) for calculating the score is represented by θ _i , the function f ′ _i (•) for classification is

It is expressed. The function f _i (•) for calculating the score is generally called a decision function. In the classification method based on supervised learning, seek a decision function f _i (·) using the data for learning that you know belongs to either advance class. That is, it is assumed that tagging is performed on the acoustic signal data (media data) for learning. In the following, this learning sound material (media data)

And is called a tagged training specimen, or simply a tagged specimen. N _T : = | D ^(T) | represents the number of tagged training samples. When learning the classifier for the i-th tag, information on all tags

Is used, and indicates whether the i-th tag is attached

Use only.

<Supervised dimension reduction method>
When learning a classifier for each tag, only information about whether the tag is attached to media data can be used. Actually, there is a correlation between different tags. For example, tags such as “rain” are often given at the same time as tags such as “storm”, “wind”, and “thunder”, but “synthesizers” and “ It is not often given at the same time as a tag such as “Loop Music”. If such tag co-occurrence information is used, the tagging accuracy may be further improved. Tag co-occurrence information can be used by handling tags at once rather than individually, but the problem is complicated when trying to learn a function that outputs all tags at once or not become. Therefore, in the present invention, a dimension reduction method that can handle all tags at once, canonical correlation analysis (CCA: Canonical Correlation Analysis, literature: H. Hotelling, “Analysis of complex variables into principal components”, J. Educ. Psych., Vol.24, 1933.), the tag co-occurrence information is used.

CCA is a technique for obtaining a mapping that maximizes the correlation of a set of two random variables. In the problem we are dealing with, since the two random variables correspond to the feature vector and the tag vector, we maximize the correlation between the (w ^CCA _x ) ^T x and (w ^CCA _y ) ^T y that map them Such a mapping w ^CCA _x , w ^CCA _y is obtained. Here, w ^CCA _x is a dim _x dimension equal to the feature vector, and w ^CCA _y is a dim _y : = | V | dimension vector equal to the tag vector. The correlation ρ between (w ^CCA _x ) ^T x and (w ^CCA _y ) ^T y is given as follows.

Where S _xx = Σ ^NT _{j = 1} x _j x ^T _j and S _yy = Σ ^NT _{j = 1} y _j y ^T _j are the self-distribution of the acoustic feature vector and the tag vector, S _xy = Σ ^NT _{j = 1} x _j y ^T _j is the covariance of the acoustic feature vector and the tag vector. Further, an average vector of x and y is assumed to be a 0 vector. Otherwise, the average vector is set to 0 by subtracting the average vector from each vector in advance. Since the value of ρ does not depend on the scale of the mappings w ^CCA _x , w ^CCA _y , by scaling appropriately, the problem of maximizing ρ is

It becomes. If this is further converted into a Lagrange dual form and the value obtained by partial differentiation with w ^CCA _x and w ^CCA _y is set to 0, the mapping that gives the maximum correlation is equivalent to the solution of the generalized eigenvalue problem below.

However, S _yx = S ^T _xy .

This eigenvalue problem can be obtained by solving min (dim _x , dim _y ) × 2 eigenvectors. However, they cannot all be used as mappings that maximize correlation. If one eigenvector and _{^{(w CCA (1) x,}} w CCA (1) y), w CCA (1) x the -w ^{CCA (1)} vector was replaced by ^{_{x (-w CCA (1) x}} , It can be easily shown from equation (8) that w ^{CCA (1)} _y ) is also an eigenvector. These two pairs of eigenvectors have positive and negative eigenvalues having the same absolute value. A positive eigenvalue is a mapping that maximizes a positive correlation, and a negative eigenvalue is a mapping that maximizes a negative correlation. Therefore, only min (dim _x , dim _y ) mappings corresponding to eigenvectors having positive eigenvalues can be used for dimension reduction. U = (w ^{CCA (1)} _x , ..., w ^{CCA (min (dimx, dimy))} _x ) is a matrix in which the mapping obtained as a solution of CCA is arranged from those corresponding to eigenvectors with large eigenvalues. Put. If this matrix is used as a transformation matrix as described below, dimension reduction can be performed.

Here, the vector after the dimension reduction is an acoustic feature vector x ′ _j, and the number of dimensions is min (dim _x , dim _y ).

<Classifier based on supervised learning>
A classification method based on supervised learning applied to automatic tagging of media data is described. The first method is a support vector machine (SVM). SVM has also been successfully applied to issues such as music genre classification and tone recognition. The second method is the Least-Squares Probabilistic Classifier (LSPC), a probabilistic classifier (literature: M. Sugiyama, “Superfast-Trainable Multi-Class Probabilistic Classifier by Least-Squares Posterior Fitting”). , in IEICE Transactions on Information and Systems, Vol. E93-D, pp. 2690-2701, 2010. Revised on June 26, 2011.). LSPC is a probabilistic classification method proposed in recent years, and since a solution is obtained analytically when learning is performed, learning can be performed at high speed and classification accuracy is also good.

<Support Vector Machine (SVM)>
SVM is a binary classification method that learns a decision function using the concept of margin. The margin is the feature space from the hyperplane that separates the positive class (y _{j, i} = 1) and the negative class (y _{j, i} = 0) to the nearest positive or negative training sample. The distance at. SVM learns the decision function to maximize the width of this margin.

  SVM uses the following linear model as a decision function.

Here, φ _SVM (•) is a mapping to some feature space. The u _SVM that maximizes the margin is known to be given in the form u _SVM = Σ ^NT _{j = 1} α _j φ _SVM (x _j ).

Can be described specifically. Here, the kernel function κ (•) corresponding to φ _SVM (•) was used. Since the threshold for classifying SVM is set to θ _i = 0, the decision boundary is a hyperplane represented by f ^SVM _i (x) = 0. Accordingly, the distance from the decision function to the training sample x _j can be calculated by | f ^SVM _i (x _j ) | / || u _SVM || (where || u || represents the Euclidean norm). Furthermore all y _j is a decision function to correctly classify training _samples, when ^{_{i = 0, f SVM i <}} 0, y j, when _i = 1, f ^SVM _i> Since the 0, all training samples x (2y _{j, i} -1) with respect to ^{_{j f SVM i (x j)}} > 0 holds. So the margin width is

It becomes. This value is independent of the scale of the _{u SVM, (2y j, i} -1) if the minimum value of f ^SVM _i (x _j) is scaling of u _SVM to be 1, the maximum width of the margin Is equivalent to maximizing 1 / || u _SVM ||, ie, minimizing || u _SVM || ² . However, the general classification problem has an overlap between two classes in the feature space, so it is not possible to find a decision boundary that completely separates all positive examples and negative examples, or even if it can be obtained May become too complex, and the generalization performance of the classifier may decrease. Therefore, the margin boundary is defined in two hyperplanes where f ^SVM _i (x) = ± 1, and the training sample where (2y _{j, i} −1) f ^SVM _i (x _j ) <1, ie, inside the margin or It is common to allow a misclassification by penalizing training samples that are on the wrong side of the decision boundary. The SVM in this case is called a soft margin SVM. As a penalty, Hinge loss ξ _j = max {0,1− (2y _{j, i} −1) f ^SVM _i (x _j )} equal to the distance from the margin boundary on the correct side is used. The optimization problem of the soft margin SVM with the penalty term added is as follows.

  However, C is a hyper parameter that determines the size of the margin and the specific gravity of the penalty.

This optimization problem is expressed in Lagrange dual form, and the result of partial differentiation with u _SVM and b is set to 0. Finally, the following optimization problem can be obtained.

_{Here, β j: = (2y j} , i -1) α j is a parameter defining the decision function, e _NT is N _T dimensional vector values of all the elements 1, matrix K ^(T) is K ⁽ (N _T × N _T ) dimensional Gram matrix defined by ^T) _{j, l} : = κ (x _j , x _l ), matrix Y has diagonal components Y _{j, j} : = 2y _{j, i} −1 (N _T × N _T ) -dimensional diagonal matrix defined by

Since Equation (14), which is an optimization problem of SVM, cannot obtain an analytical solution, optimization is performed using quadratic programming. If the optimal solution β ^* = (β ^* ₁ , ..., β ^* _NT ) ^T is determined, then determine the parameter of the decision function as α ^* _j : = (2y _{j, i} −1) β ^* _j The decision function can be calculated using equation (11).

<Least squares probabilistic classifier (LSPC)>
LSPC is a classifier based on posterior probability estimation, and estimates posterior probability using the following linear model.

_{Here, φ j, a '(·} ) is when the dimension of the feature vector x to dimx, all possible inputs ^{(x, a) ∈R dimx φ} j with respect × _{{0,1}, a'} a basis function that satisfies (x, a) ≧ 0,

Is a vector representation of all basis functions,

Is a parameter to be learned. Note that LSPC is a technique that can handle multi-class classification problems, but here the classes are assumed to be two classes with a = 0,1. In the following, as a basis function, the document: M. Sugiyama, “Superfast-Trainable Multi-Class Probabilistic Classifier by Least-Squares Posterior Fitting”, in IEICE Transactions on Information and Systems, Vol.E93-D, pp.2690-2701, 2010 Consider the following kernel model used in Revised on June 26, 2011.

  Where δ (·) is the Kronecker delta given by:

  The LSPC determines the parameter α so as to minimize the square error between the true class posterior probability and the estimated class posterior probability calculated from the following equation.

However, the (2N _T × 2N _T ) -dimensional matrix H and the 2N _T- dimensional vector h are defined as follows.

  Since the matrices H and h include unknown probability density functions p (x) and p (x, y), they are approximated using training samples as follows.

The following expression, which is obtained by approximating the square error J (α) using the approximated matrix ^ H and ^ h and further adding the normalization term λα ^T α, is the objective function of LSPC.

  However, λ is a hyperparameter that determines the size of the normalization term.

  The optimal solution of this objective function is

It becomes. Here, I _2NT is a (2N _T × 2N _T ) -dimensional unit matrix.

When the kernel model represented by equation (18) is used as the basis function, if the basis function vector φ (x, a) is divided into the upper half and the lower half, the one corresponding to the class value a is non-zero. One is a zero vector. At this time, the function value q _i (a | x; α) is calculated as shown in FIG. 2 when the parameter α is divided into two vectors having the same number of elements α (0) and α (1). Calculated using only the parameter α (a) for the class.

Furthermore, the result of the multiplication of basis function vectors φ (x, 0) used for the matrix H is such that all blocks other than the upper left divided into four as shown in FIG. 3 are zero matrices (2N _T × 2N _T ). It becomes a matrix.

  Multiplying basis function vectors φ (x, 1) with class value inputs set to 1, the non-zero block only changes from upper left to lower right, so the matrix ^ H calculated by these sums is It becomes such a block diagonal matrix.

Here, the matrix ~ H is an (N _T × N _T ) -dimensional matrix defined as follows.

Therefore, ^ _T ^(a) is an _NT dimension vector defined by

Then, the optimum solution of the parameter α ^(a) corresponding to each class can be individually solved from the following optimization problem for each class.

Here, _INT is a (N _T × N _T ) dimensional identity matrix.

The optimization of the function q _i (a | x; α) can be performed by solving equation (30) for all classes a∈ {0,1}. However, the output value of this function cannot be used as it is as an estimate of the posterior probability. This is because the posterior probability does not take a negative value, but the output of the function q _i (a | x; α) can take a negative value. Therefore, when the output value becomes a negative value, it is necessary to correct the value to 0. Further, the output of the function q _i (a | x; α) does not satisfy the property that the sum of posterior probabilities for all classes is always 1. Therefore, LSPC performs normalization so that the sum of the values of function q _i (a | x; α) estimated for all classes at the end is 1, and the estimated value of class posterior probability satisfies this property. Adjust as follows. Eventually, the decision function f ^LSPC _i (x) = ^˜P (y _{*, i} = 1 | x) when performing classification using LSPC is calculated by the following equation.

<Automatic tagging method based on semi-supervised learning>
<Formulation of problem>
Media data without tags can be easily collected, but these training samples cannot be used for learning within the framework of supervised learning. Using a method based on semi-supervised learning, a method for creating a classifier using both tagged and untagged training samples for learning is described.

The problem of automatic tagging of media data when using semi-supervised learning can be handled in almost the same framework as the formulation in the previous chapter. That is, the output y of the tag vector with respect to the input x of the feature vector is determined by treating each tag individually as shown in Equation (2). Further, whether or not to add an individual tag is determined by using the output of a decision function f _i (•) that outputs a real value as shown in Equation (5). The difference from supervised learning is that learning of the decision function f _i (•) is performed by using an untagged training sample in addition to the tagged training sample D (T). By such an extension, the learning part of FIG. 1 is changed as shown in FIG. In the following, an untagged training sample

The number is N _U : = | D ^(U) |. In addition, the total number of training samples including the tagged and untagged training samples is expressed using N: = N _T + N _U.

<Semi-supervised dimension reduction method SemiCCA>
As a supervised dimension reduction method, CCA that handles correlation between tags has been described, but it has been found that if CCA performs dimension reduction, over-learning occurs and classification performance deteriorates. Semi-supervised dimension reduction method SemiCCA to prevent such over-learning using untagged training samples (Reference: A. Kimura, H. Kameoka, M. Sugiyama, T. Nakano, E. Maeda, H. Sakano, K. Ishiguro, “SemiCCA: Efficient semi-supervised learning of canonical correlations”, in International Conference on Pattern Recognition (ICPR), pp. 2933-2936, Istanbul, Turkey, 2010.). SemiCCA is a semi-supervised dimension reduction method based on CCA proposed in recent years. SemiCCA combines principal component analysis (PCA) with CCA to perform correction based on the principal components of all tagged and untagged training samples to prevent over-learning. If the mapping of the feature vector and the tag vector onto the principal component axis is set as w ^PCA _x and w ^PCA _y , respectively, the solution of these PCAs can be obtained by solving the following eigenvalue problem.

^{_{Here, S 'xx: = Σ N}} j = 1 x j x Tj is the self-dispersing matrix of all training feature vector of tagged and untagged.

  SemiCCA solves the following generalized eigenvalue problem by combining Eq. (8) with Eq. (32) and Eq. (33), which are PCA eigenvalue problems.

  Here, D and E are matrices defined by the following equations, and η is a parameter that determines the weights of CCA and PCA.

The semi-CCA generalized eigenvalue problem is consistent with the CCA generalized eigenvalue problem when η = 1, and is consistent with the PCA eigenvalue problem when η = 0. The mapping used for dimensionality reduction with SemiCCA is the eigenvector corresponding to min (dim _x , dim _y ) largest eigenvalues equal to the number of CCA solutions. When these eigenvectors are arranged side by side and U _Semi is used, the feature vector x ′ _j after dimension reduction can be obtained as in the following Expression (37).

_< Classifier based on semi-supervised learning>
There are several classification methods that semi-supervised SVM. In the present invention, Laplacian-SVM (literature: M. Belkin, P. Niyogi, and V. Sindhwani, “On Manifold Regularization”, in Proceedings of the 10th International Workshop on Artificial Intelligence and Statistics (AISTATS), Barbados, January 2005.). Laplacian-SVM is SVM semi-supervised by Laplacian normalization. This semi-supervised method is a semi-supervised method of adding a term called a Laplacian normalization term that makes the function smoother than the objective function.

<Laplacian-SVM>
Laplacian normalization is a semi-supervised technique that smooths the decision function using tagged and untagged training samples. The operation that is specifically performed with the semi-teaching is to add a penalty term based on the smoothness of the function to the objective function of the optimization problem to be solved when determining the parameters of the decision function of the classifier. The smoothness of the decision function f ^LapSVM _i can be expressed by the following S _LapSVM , and the function becomes smoother as this value approaches zero.

Where W _{j, l} is a weight determined based on the distance between samples x _j and x _l ,

Is a vector representation of the decision function output values for all training samples. In addition, L is a matrix called Graph Laplacian, and the weight W _{j, l} in the matrix notation is W, and the (N × N) dimensional matrix D is represented by D _{j, j} : = Σ _l W _{j, l} When defined diagonal matrix is defined as L: = D−W. Expression (38) has a form in which the square of the difference between the output values of the decision functions for all pairs of training samples is added with a weight. Therefore, if the weight W _{j, l} is set to a large value for a pair of samples with a short distance and a small value for a sample pair with a long distance, the output value of the decision function for a sample with a short distance is close and smooth. It can be said that the value of S _LapSVM decreases as the function increases. It should be noted in equation (38) that not only tagged training samples but also untagged training samples are used in this calculation.

In Laplacian-SVM, since the decision function is learned so that the value of S _lapSVM becomes small, optimization is performed after adding this value to the equation (13) of the objective function of SVM. Ie

Is the objective function of Laplacian-SVM. Here, γ is a hyperparameter that determines the size of the Laplacian normalization term. By changing the objective function in this way, the expression using the kernel of the optimal decision function is not in the form like equation (11), but the function with the kernel placed on the untagged training sample

It will be represented by

  Further, when the same operation as that of SVM is performed to transform equation (40), the following objective function is obtained.

Where the (N × N) dimensional matrix K is the Gram matrix of all tagged and untagged training samples, and the (N _T × N) dimensional matrix B is for all j∈ {1, ..., N _T } B _{j, j} = 1 and all other elements are zero. Using the N-dimensional vector β ^{Lap *} which is the solution, the parameter α ^* = (α ^* ₁ ,... Α ^* _N ) ^T of the decision function f ^LapSVM _i (•) is calculated as follows.

Since equation (42) of the optimization problem that can not be obtained analytical solutions as for formula (14) is a SVM optimization problem, alpha ^* vector used to calculate beta ^{Lap *} etc. quadratic programming It is necessary to obtain using

[First Embodiment]
<System configuration>
FIG. 5 is a block diagram showing the media data analysis apparatus 100 according to the first embodiment of the present invention. The media data analyzing apparatus 100 includes a tagged learning data set that is a set of media data to which tag information that is information for describing media data is assigned in advance, and a tag that is a set of media data to which no tag information is given. None Input a learning data set, learn a classifier to classify tag information to be given to media data, input media data to which no tag information is given, and output tag information to be given to media data Specifically, it is composed of a computer that includes a CPU (Central Processing Unit), a RAM, and a ROM that stores a program for executing a classifier learning processing routine and a tagging processing routine described later. Functionally, it is configured as follows.

  The media data analysis apparatus 100 includes an input unit 10, a calculation unit 20, and a tag vector output unit 30.

  The input unit 10 includes a tagged learning data set, which is a set of media data to which tag information, which is information describing the media data, is given in advance, and untagged learning, which is a set of media data to which no tag information is given. In addition to receiving the input of the data set for the media, the input of the media data to which no tag information is given is received.

  The calculation unit 20 includes a learning database 1, a learning data feature extraction unit 2, a dimension reduction space learning unit 3, a classifier learning unit 4, an input data feature extraction unit 5, a feature dimension reduction unit 6, and a classifier evaluation unit 7. It has. The classifier learning unit 4 is an example of a classification model learning unit, and the classifier evaluation unit 7 is an example of a classification unit.

  The learning database 1 is a database in which tagged media data and untagged media data are accumulated, and stores an input tagged learning data set and an untagged learning data set.

The learning data feature extraction unit 2 calculates a feature vector from the media data of the learning data. In the following description, for the sake of simplicity, the description is limited to the case where an acoustic signal is employed as an example of media data. By appropriately changing the feature extraction method, the present invention can be widely applied to media data other than sound signals, for example, image signals, video signals, texts, microblogs, and the like. When the target is an acoustic signal, specifically, the acoustic signal is subjected to frame analysis, and from each frame, Mel-Frequency Cepstrum Coefficients (MFCC: Literature: P. Mermelstein “DistanceMeasure for Speech Recognition, Psychological and Instrumental ”, in Pattern Recognition and Artificial Intelligence, pp. 374-388, June 1976.). MFCC is an acoustic feature that takes human auditory characteristics into account, and is used as a standard in the field of music information retrieval. Furthermore, since the acoustic signal of the sound material changes with time, ΔMFCC, which is an approximate value of the time derivative of MFCC, and ΔMFCC, which is an approximate value of the secondary derivative, are calculated as dynamic feature quantities. Next, the local feature amounts composed of the MFCC, ΔMFCC, and ΔΔMFCC calculated for each frame are collected to create a bag-of-features feature amount, which is used as a feature vector (acoustic feature vector). To create bag-of-features features, first collect local features extracted from all sound materials and create a codebook using the LBG algorithm (Fig. 6). Then, all local feature values extracted from one sound material are vector quantized using a code book, and a histogram thereof is created. This is normalized bag-of-features and used as a feature vector. The bag-of-features feature is used not only in the image field but also often in the tune recognition field. On the other hand, if the i-th tag v _i is given as a tag assigned to the sound material, the value y _{j, i} of the i-th element of the vector y _j is set to 1, and 0 is not given. And create a tag vector.

The dimension reduction space learning unit 3 solves the generalized eigenvalue problem of the above equation (34) using the feature vector x _j calculated from each media data of the learning data including the media data not tagged. Then, a transformation matrix U _Semi is constructed, and a feature vector x ′ _j is created for each media data of the learning data by the above equation (37).

The classifier learning unit 4 creates a classifier by LSPC for each tag. Here, in the learning database 1, the classifier is learned using the tagged media data. Specifically, for each tag, based on the feature vector x ′ _j created for each media data tagged with learning data, α ^{(a) *} in the above equation (30) is changed to all classes a∈. Calculate for {0,1}. The kernel function

Use a Gaussian kernel.

Similar to the learning data feature extraction unit 2, the input data feature extraction unit 5 creates a feature vector x _q from the input media data.

The feature dimension reduction unit 6 uses the transformation matrix U _Semi created by the dimension reduction space learning unit 3 to reduce the dimension of the feature vector x _q as an input and create x ′ _q .

The classifier evaluation unit 7 calculates the decision function of the above equation (31) for each tag using the LSPC parameter α created for each tag by the classifier learning unit 4. If this is greater than 0.5, the i-th tag v _i is assigned. This is performed for all I pieces of LSPC classifier inputs 0 or 1 in the i-th element of the tag vector y _q.

The generated tag vector y _q outputs by the tag vector output section 8.

<Operation of media data analyzer>
Next, the operation of the media data analysis apparatus 100 according to this embodiment will be described. First, when a tagged learning data set to which a tag vector is given and an untagged learning data set to which a tag vector is not given are input to the media data analyzing apparatus 100, the media data analyzing apparatus 100 inputs them. The tagged learning data set and untagged learning data set are stored in the learning database 1. Then, the media data analysis apparatus 100 executes a classifier learning process routine shown in FIG.

First, in step S101, a feature vector is extracted from each learning data in the tagged learning data set and the untagged learning data set, and a tag vector is created for each learning data in the tagged learning data set. . In step S102, based on the feature vectors of the learning data in the tagged learning data set and the untagged learning data set extracted in step S101, the above equation (34) is solved to obtain a transformation matrix. Construct U _semi .

In step S103, the transformation matrix U _semi configured in step S102 is applied to the feature vectors of the learning data in the tagged learning data set and the untagged learning data set extracted in step S101. Thus, a feature vector after dimension reduction is created for each learning data in the tagged learning data set and the untagged learning data set.

In the next step S104, a classifier by LSPC is learned for each tag based on the feature vector after dimension reduction of each learning data of the tagged learning data set created in step S103, and the above equation Α ^{(a) *} of (30) is obtained, and the classifier learning processing routine is terminated.

  When an untagged acoustic signal is input to the media data analysis apparatus 100, the media data analysis apparatus 100 executes a tag addition processing routine shown in FIG.

  In step S111, the input media data is received. In step S112, a feature vector is extracted from the input media data.

In the next step S113, the feature vector extracted in the step S112 is applied to the transformation matrix U _semi created in the classifier learning processing routine to create a feature vector after dimension reduction.

Then, in step S114, for each tag, the determination function of the above equation (31) is calculated based on α ^{(a) *} obtained for the tag and the dimension-reduced feature vector created in step S113. To do. In step S115, depending on whether the value of the decision function calculated in step S114 is greater than 0.5 for each tag, a tag vector is generated by assigning the tag, and the tag vector output unit 30 outputs the tag vector. The tag assignment processing routine is terminated.

  As described above, according to the media data analysis apparatus according to the first embodiment, based on the feature vectors extracted for each media data of the tagged learning data set and the untagged learning data set, Based on posterior probability estimation, learning a matrix that represents a mapping to reduce the number of dimensions of a feature vector, creating a feature vector with reduced dimensions for the feature vector of each media data in the tagged learning data set Learn classifier. A feature vector with a reduced dimension is created for the feature vector of the input media data, and based on the learned classifier, it is classified for each tag of the tag vector whether or not the tag is assigned. Thereby, tag information can be given with high accuracy even in a situation where learning data of a sufficient scale cannot be obtained.

[Second Embodiment]
<System configuration>
Next, a second embodiment of the present invention will be described. Note that the configuration of the media data analysis apparatus according to the second embodiment is the same as that of the first embodiment, and thus the same reference numerals are given and description thereof is omitted.

  In the second embodiment, the fact that a transformation matrix for dimension reduction is learned only from tagged learning data and that a tag is assigned using an SVM classifier are This is different from the first embodiment.

  The learning database 1 of the media data analyzing apparatus according to the second embodiment is a database in which tagged media data is accumulated, and stores an input tagged learning data set.

  The learning data feature extraction unit 2 calculates a feature vector from each media data of the learning data and creates a tag vector.

The dimension reduction space learning unit 3 solves the generalized eigenvalue problem of the above equation (8) using the feature vector x _j calculated from each media data of the learning data, forms a transformation matrix U, A feature vector x ′ _j is created by Expression (9).

The classifier learning unit 4 creates a SVM classifier for each tag. Specifically, the optimal solution β ^* = (β ^* ₁ ,..., Β ^* _NT ) ^T of Equation (14), which is an SVM optimization problem, is obtained using quadratic programming. The kernel function

Use a Gaussian kernel.

As in the first embodiment, the input data feature extraction unit 5 performs frame analysis on input media data (for example, an acoustic signal), and calculates MFCC, ΔMFCC, and ΔMFCC from each frame. Then, by using the codebook created in the learning data characteristic extraction unit 2, MFCC, ΔMFCC, to create a feature vector x _q and vector quantizing the feature amount of DerutaderutaMFCC. The feature dimension reduction unit 6 uses the transformation matrix U created by the dimension reduction space learning unit 3 to reduce the dimension of the feature vector x _q as an input and create x ′ _q .

The classifier evaluation unit 7 calculates α ^* _j : = (2y _{j, i} −1) β ^* _j using the SVM optimum solution β ^* created by the classifier learning unit 4, and the input feature vector Calculate the decision function of equation (11) above for x ′ _q . That is, it is classified whether or not the i-th tag v _i is assigned. This is performed for all I number of classifiers, the i th element of the tag vector y _q Type 0 or 1, and generates a tag vector y _q.

The generated tag vector y _q outputs by the tag vector output section 8.

  Note that other configurations and operations of the media data analysis apparatus according to the second embodiment are the same as those of the first embodiment, and thus description thereof is omitted.

[Third Embodiment]
<System configuration>
Next, a third embodiment of the present invention will be described. Note that the configuration of the media data analysis apparatus according to the third embodiment is the same as that of the first embodiment, and therefore the same reference numerals are given and description thereof is omitted.

  The third embodiment is different from the second embodiment in that tag vectors are assigned using a classifier based on LSPC.

  The learning database 1 of the media data analysis apparatus according to the third embodiment is a database in which tagged media data is accumulated, as in the second embodiment.

The learning data feature extraction unit 2 calculates a feature vector from each piece of media data of learning data and creates a tag vector, as in the second embodiment. Similar to the second embodiment, the dimension reduction space learning unit 3 solves the generalized eigenvalue problem of the above equation (8) to construct the transformation matrix U, and the above equation (9) A feature vector x ′ _j is created.

The classifier learning unit 4 creates a classifier by LSPC for each tag. Specifically, for each tag, α ^{(a) *} in the above equation (30) is calculated for all classes a∈ {0, 1}. The kernel function

Use a Gaussian kernel.

The input data feature extraction unit 5 creates a feature vector x _q from the input media data, as in the first embodiment.

The feature dimension reduction unit 6 uses the transformation matrix U created by the dimension reduction space learning unit 3 to reduce the dimension of the feature vector x _q as an input and create x ′ _q .

The classifier evaluator 7 calculates the decision function of the above equation (31) for each tag using the LSPC parameter α created for each tag by the classifier learning unit 4. If this is greater than 0.5, the i-th tag v _i is assigned. This is performed for all I pieces of LSPC classifiers, the i th element of the tag vector y _q Type 0 or 1, and generates a tag vector y _q.

The generated tag vector y _q outputs by the tag vector output section 8.

  Note that other configurations and operations of the media data analysis apparatus according to the third embodiment are the same as those of the first embodiment, and thus the description thereof is omitted.

[Fourth Embodiment]
<System configuration>
Next, a fourth embodiment of the present invention will be described. Note that the configuration of the media data analysis apparatus according to the fourth embodiment is the same as that of the first embodiment, and therefore the same reference numerals are given and description thereof is omitted.

  The fourth embodiment is different from the second embodiment in that tag vectors are assigned using a Laplacian-SVM classifier.

  The learning database 1 of the media data analysis apparatus according to the fourth embodiment is a database in which tagged media data and untagged media data are stored.

The learning data feature extraction unit 2 calculates a feature vector from each piece of media data of learning data and creates a tag vector, as in the second embodiment. Similar to the second embodiment, the dimension reduction space learning unit 3 solves the generalized eigenvalue problem of the above equation (8) to construct the transformation matrix U, and the above equation (9) A feature vector x ′ _j is created.

The classifier learning unit 4 creates a Laplacian-SVM classifier for each tag. Specifically, the optimal solution β ^{Lap *} of Equation (42), which is a Laplacian-SVM optimization problem, is obtained using quadratic programming. The kernel function

Use a Gaussian kernel.

Similar to the learning data feature extraction unit 2, the input data feature extraction unit 5 creates a feature vector x _q from the input media data.

The feature dimension reduction unit 6 uses the transformation matrix U created by the dimension reduction space learning unit 3 to reduce the dimension of the feature vector x _q as an input and create x ′ _q .

The classifier evaluator 7 calculates α ^* in Equation (43) using the Laplacian-SVM optimal solution β ^{Lap *} created for each tag by the classifier learning unit 4, and inputs features for each tag. Calculate the decision function of equation (41) for the vector x ′ _q . That is, it is classified whether or not the i-th tag v _i is assigned. This is performed for all I number of classifiers, the i th element of the tag vector y _q Type 0 or 1, and generates a tag vector y _q.

The generated tag vector y _q outputs by the tag vector output section 8.

  Note that other configurations and operations of the media data analysis apparatus according to the fourth embodiment are the same as those of the first embodiment, and thus description thereof is omitted.

[Fifth Embodiment]
<System configuration>
Next, a fifth embodiment of the present invention will be described. Note that the configuration of the media data analysis apparatus according to the fifth embodiment is the same as that of the first embodiment, and therefore the same reference numerals are given and description thereof is omitted.

  The fifth embodiment is different from the first embodiment in that a tag vector is assigned using a classifier based on SVM.

  The learning database 1 of the media data analysis apparatus according to the fifth embodiment is a database in which tagged media data and untagged media data are stored. As in the first embodiment, the learning data feature extraction unit 2 calculates a feature vector from each media data of the learning data and creates a tag vector for each tagged media data. .

The dimension reduction space learning unit 3 solves the generalized eigenvalue problem of the above equation (34), forms a transformation matrix U _Semi , and creates a feature vector x ′ _j by the above equation (37).

The classifier learning unit 4 creates a SVM classifier for each tag. Here, in the learning database 1, the classifier is learned using the tagged media data. Specifically, the optimal solution β ^* = (β ^* ₁ ,..., Β ^* _NT ) ^T of Equation (14), which is an SVM optimization problem, is obtained using quadratic programming. The kernel function is

Use a Gaussian kernel.

Similar to the learning data feature extraction unit 2, the input data feature extraction unit 5 creates a feature vector x _q from the input media data.

The feature dimension reduction unit 6 uses the transformation matrix U _Semi created by the dimension reduction space learning unit 3 to reduce the dimension of the feature vector x _q as an input and create x ′ _q .

The classifier evaluation unit 7 calculates α ^* _j : = (2y _{j, i} −1) β ^* _j using the SVM optimal solution β ^* created for each tag by the classifier learning unit 4, For each tag, the decision function of equation (11) is calculated for the input feature vector x ′ _q . That is, it is classified whether or not the i-th tag v _i is assigned. This is performed for all I number of classifiers, the i th element of the tag vector y _q Type 0 or 1, and generates a tag vector y _q.

The generated tag vector y _q outputs by the tag vector output section 8.

  Note that other configurations and operations of the media data analysis apparatus according to the fifth embodiment are the same as those of the first embodiment, and thus description thereof is omitted.

[Sixth Embodiment]
<System configuration>
Next, a sixth embodiment of the present invention will be described. Note that the configuration of the media data analysis apparatus according to the sixth embodiment is the same as that of the first embodiment, and thus the same reference numerals are given and description thereof is omitted.

  The sixth embodiment is different from the first embodiment in that tag vectors are assigned using a Laplacian-SVM classifier.

The learning database 1 of the media data analysis apparatus according to the sixth embodiment is a database in which tagged media data and untagged media data are stored. As in the first embodiment, the learning data feature extraction unit 2 calculates a feature vector from each media data of the learning data and creates a tag vector for each tagged media data. . The dimension reduction space learning unit 3 solves the generalized eigenvalue problem of the above equation (34), forms a transformation matrix U _Semi , and creates a feature vector x ′ _j by the above equation (37).

The classifier learning unit 4 creates a Laplacian-SVM classifier for each tag. Specifically, the optimal solution β ^{Lap *} of Equation (42), which is a Laplacian-SVM optimization problem, is obtained using quadratic programming.

The kernel function

Use a Gaussian kernel.

Similar to the learning data feature extraction unit 2, the input data feature extraction unit 5 creates a feature vector x _q from the input media data.

The feature dimension reduction unit 6 uses the transformation matrix U _Semi created by the dimension reduction space learning unit 3 to reduce the dimension of the feature vector x _q as an input and create x ′ _q .

The classifier evaluator 7 calculates α ^* in Equation (43) using the Laplacian-SVM optimal solution β ^{Lap *} created for each tag by the classifier learning unit 4, and inputs features for each tag. Calculate the decision function of equation (41) for the vector x ′ _q . That is, it is classified whether or not the i-th tag v _i is assigned. This is performed for all I number of classifiers, the i th element of the tag vector y _q Type 0 or 1, and generates a tag vector y _q.

The generated tag vector y _q outputs by the tag vector output section 8.

  Note that other configurations and operations of the media data analysis apparatus according to the sixth embodiment are the same as those of the first embodiment, and thus description thereof is omitted.

[Seventh Embodiment]
<System configuration>
Next, a seventh embodiment of the present invention will be described. In addition, about the part which becomes the structure similar to 1st Embodiment, the same code | symbol is attached | subjected and description is abbreviate | omitted.

  The seventh embodiment is different from the second embodiment in that feature vector dimension reduction is not performed.

  As shown in FIG. 9, the computing unit 720 of the media data analysis apparatus 700 according to the seventh embodiment of the present invention includes a learning database 1, a learning data feature extraction unit 2, a classifier learning unit 704, input data. A feature extraction unit 5 and a classifier evaluation unit 707 are provided.

  The learning database 1 is a database in which tagged media data is accumulated, and stores an input tagged learning data set.

The learning data feature extraction unit 2 calculates a feature vector x _j from each media data of the learning data and creates a tag vector.

The classifier learning unit 704 creates an SVM classifier for each tag. Specifically, using quadratic programming, based on the feature vector x _j of each learning data, the optimal solution β ^* = (β ^* ₁ ,. .., β ^* _NT ) ^T is obtained.

The input data feature extraction unit 5 creates a feature vector x _q from the input media data, as in the first embodiment.

The classifier evaluation unit 707 calculates α ^* _j : = (2y _{j, i} −1) β ^* _j using the SVM optimum solution β ^* created by the classifier learning unit 4, and the input feature vector Calculate the decision function of equation (11) above for x _q . That is, it is classified whether or not the i-th tag v _i is assigned. This is performed for all I number of classifiers, the i th element of the tag vector y _q Type 0 or 1, and generates a tag vector y _q.

The generated tag vector y _q outputs by the tag vector output section 8.

  Note that the other configuration and operation of the media data analysis apparatus according to the seventh embodiment are the same as those of the first embodiment, and thus the description thereof is omitted.

  In the seventh embodiment, the case of learning a classifier based on SVM has been described as an example. However, the present invention is not limited to this, and the classifier based on LSPC is not limited to this. And a tag vector may be assigned using a classifier based on LSPC. Further, as in the fourth embodiment, a classifier based on Laplacian-SVM may be learned, and a tag vector may be assigned using a classifier based on Laplacian-SVM.

<Experimental result>
A classification method and a dimension reduction method based on supervised learning and semi-supervised learning were applied to the automatic tagging problem of media data, respectively.

  Here, experimental conditions will be described. The sound material data of Freesound (http://www.freesound.org/) was used as media data for experiments with automatic tagging. Freesound is a web service that shares music materials based on the Creative Commons License (http://creativecommons.org/), which allows secondary use of copyrighted works under conditions specified by the creator. With this web service, anyone who registers as a user can register and share music materials created or recorded by himself, and many sound materials are already registered. Many of the registered music materials are street sounds recorded by field recording, natural sounds such as mountains and rain, door opening and closing sounds and machine operating sounds. In addition to this, there are many short materials for loop music composed by synthesizers. In addition, when registering a sound material, a plurality of arbitrary words can be assigned to the material as tags. Therefore, there are a wide variety of search tags on Freesound.

  During the experiment, we used 2012 audio material with WAV format, sampling rate of 44.1kHz, and quantization bit rate of 16bit from this Freesound database. For stereo files, the sound of both channels was averaged and converted to a monaural signal.

The frame for extracting the MFCC feature value is 23 milliseconds, and each frame is divided by 12.5 milliseconds so that it overlaps with each of the previous and next frames. In addition, the MFCC feature values extracted from each frame are coefficients up to the first 13 dimensions. Therefore, the dimension number of the local feature amount including the Δ component and the ΔΔ component is 39. In the case of vector quantization clustering, if all local features are used, it takes a lot of computation time, so clustering was performed by collecting random ramping of 500 local features from all acoustic materials. . For clustering, a standard LBG algorithm was used as a vector quantization method, and the number of clusters was 2048. The number of dimensions of the bag-of-features feature is equal to the number of clusters, and dim _x = 2048.

In 2012, Freesound's 2012 sound materials were tagged with more than 1000 types of tags, but most of these tags were only given to a small number of music materials, so the number of music materials attached. Only 230 tags with 12 or more were used for the experiment. That is, dim _y = | V | = 230. Table 1 shows an example of these tags.

  In the experiment, out of 2012 sound materials, 1000 tagged training samples, 712 untagged training samples, and 300 evaluation samples were randomly divided and used. It should be noted that at least four of the 1000 tagged training samples and at least one of the 300 evaluation samples were included in the sound material to which each tag was assigned. The performance evaluation of the classification method was carried out using the average value of AUC (Area Under the Curve), in which each sample was randomly selected 50 times by the above-mentioned sample division method, and the experiment was performed on all of them. AUC is the area under the receiver operating characteristic curve (ROC curve). The ROC curve is a curve as shown in FIG. 10, and is drawn by the numbers of positive examples and negative examples classified into the positive class when the threshold value is changed from a large value to a small value. Obviously, using a decision function that outputs a large value for positive examples and a small value for negative examples provides better classification performance, so a classifier with a higher AUC value has better classification performance. I can say that. Since SVM learns the decision function after setting the threshold to 0, the threshold at the time of classification is originally fixed to 0, but when using AUC, the threshold is moved to improve its performance. Care should be taken to evaluate. First, we apply classification and dimension reduction techniques based on supervised learning to the problem of automatic tagging. Kernel Density Estimation (KDE), literature: M. Wang, X. Hua, T. Mei, R. Hong, G. Qi, Y. Song, and L. Dai, “Semi-supervised Kernel density estimation for video annotation ”, Computer Vision and Image Understanding, Vol. 113, No. 1, pp. 384-396, January 2009.).

  All classification methods based on SVM, LSPC, and KDE contain kernel functions inside. Here is a commonly used Gaussian kernel for this kernel function

Was used. Here, σ is a hyperparameter called kernel width. Moreover, the value of the hyperparameter included in each classification method was determined by the 2-fold intersection confirmation method from the following candidates (1) to (3).

(1) The candidate of Gaussian kernel width σ, which is the hyperparameter of all three methods, is {m / 20, m / 10, m / 6, m / 3, m / 2, m, 5m / 2,5m, 10m }. Where m is the median distance between all training samples.

(2) The candidate values for the coefficient λ of the normalization term, which is a hyperparameter of LSPC, are {10 ⁻¹⁰ , 10 ⁻⁶ , 10 ⁻⁴ , 10 ⁻² , 1}.

(3) {1 / m ′, 0.5,1,10} is used as a candidate for hyperparameter C that determines the tradeoff between the SVM margin size and Hinge loss. Here, m 'is the inverse of the average value of || x _j || ^2.

In classification using supervised learning, experiments were conducted by two methods, one using these parameters in common for all tag classifiers and the other using different values for each classifier for each tag. The feature vectors used for classification were the bag-of-features features as they were and the features that were dimension-reduced by CCA. The library SVM ^light (literature: T. Joachims, “SVMlight: Support Vector Machine”, http://svmlight.joachims.org/, University of Dortmund, November 1999.) was used for SVM learning.

  Table 2 shows the experimental results of the classification method based on supervised learning.

  Comparing the case where dimensionality reduction is performed with CCA and the case where dimensionality reduction is not performed, it can be seen that performance is degraded when dimensionality reduction is applied under most conditions. This is a result contrary to the initial expectation that the classification performance is improved by performing dimension reduction. On the other hand, a major cause of the drop in the classification performance due to the reduction in dimensions is that there are too few training samples with tags, resulting in overlearning.

  Next, instead of CCA used for dimension reduction, experiments were performed using a semi-supervised dimension reduction technique SemiCCA. The SVM and LSPC hyperparameters are common to all classifiers, and the KDE kernel width is a different value for each tag. In addition, the parameter η of SemiCCA is uniformly η = 0.99.

  Table 3 summarizes the results.

  From this result, it can be seen that the classification performance using the feature vector subjected to dimension reduction by SemiCCA is always better than the classification performance when dimension reduction is performed by CCA. When these two classification performances were subjected to a t-test with a significance level of 5%, it was shown that there was actually a significant difference. In particular, LSPC and KDE achieved significantly better results when dimensionality reduction was performed with SemiCCA than when dimensionality reduction was not performed, and succeeded in improving classification performance by dimensionality reduction. On the other hand, when SVM is used, classification performance is worse when dimension reduction is performed with SemiCCA than before dimension reduction.

  Finally, we apply Laplacian-SVM, a semi-supervised classification method, to the auto-tagging problem. Conventional methods include semi-supervised kernel density estimation (SSKDE: Kernel Density Estimation, literature: M. Wang, X. Hua, T. Mei, R. Hong, G. Qi, Y. Song, and L. Dai, “ Semi-supervised kernel density estimation for video annotation ”, Computer Vision and Image Understanding, Vol. 113, No. 1, pp. 384-396, January 2009.).

  Tagged and untagged training specimens and evaluation specimens used for experiments shall be the same as before. The kernel function used in each method is the same Gaussian kernel as the supervised classifier. The weight matrix W used to calculate Graph Laplacian in Laplacian-SVM is

The binary weight defined in (1) was used. In addition, the reliability value τ of the tag information of SSKDE is 0.95 which is often used. Other hyperparameters were determined from the following candidates (1) to (5) using the 2-fold intersection confirmation method.

(1) SSKDE kernel widths are {m / 20, m / 10, m / 6, m / 4, m / 3, m / 2, m, 5m / 2, 5m, 10m} as candidates.

(2) The Laplacian-SVM kernel width was selected as {m / 4, m / 3, m / 2}, which was selected in the supervised SVM.

(3) The weight matrix W in Laplacian-SVM and the number of neighbors k of the sparse adjacency matrix M ′ of SSKDE are {1, 5, 10, 20, 30}, respectively.

(4) The coefficient γ of the Laplacian normalization term of Laplacian-SVM is {N / 100, N / 10, N / 2} as candidates.

(5) The parameter C that determines the trade-off between Laplacian-SVM margin width and Hinge loss is {0.5, 1, 10}.

  As a dimension reduction technique, SSKDE uses SemiCCA, which showed the best results in each semi-supervised technique, and Laplacian-SVM reduced classification performance when dimension reduction was performed in semi-supervised SVM. Therefore, the feature vector is used as it is without any dimension reduction. Table 4 summarizes the results.

  It can be said that the classification performance of SSKDE is greatly improved compared to the supervised classification method. On the other hand, the classification performance of Laplacian-SVM was almost the same as that of supervised SVM, and the t-test with a significance level of 5% showed no significant difference.

  Note that the present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

  For example, the media data may be other than sound material (acoustic signal). For example, a tag may be attached to content data such as image data, video data, text data, and the like.

  In the present specification, the embodiment has been described in which the program is installed in advance. However, the program can be provided by being stored in a computer-readable recording medium.

DESCRIPTION OF SYMBOLS 1 Learning database 2 Learning data feature extraction part 3 Dimension reduction space learning part 4, 704 Classifier learning part 5 Input data feature extraction part 6 Feature dimension reduction part 7, 707 Classifier evaluation part 8 Tag vector output part 10 Input part 20, 720 Calculation unit 30 Tag vector output unit 100, 700 Media data analysis device

Claims (3)

A media data analysis device for giving tag information, which is information for explaining the content data, to given content data,
Content data that is an element of a tagged learning data set that is a set of content data to which the tag information is assigned in advance, and content that is an element of an untagged learning data set that is a set of content data to which no tag information is given Data feature extraction means for learning for extracting a feature vector that is a vector expressing the characteristics of content data from each of the data;
Based on the feature vector extracted for each content data of the tagged learning data set and the pre-assigned tag information, and the feature vector extracted for each content data of the untagged learning data set. Dimensional reduction space learning means for learning a matrix representing a mapping for reducing the number of dimensions of the feature vector;
Dimensions for generating the feature vectors with reduced dimensions from each of the feature vectors extracted for each content data of the tagged learning data set based on the matrix learned by the dimension reduction space learning means. Reduction feature generation means;
Tag information to be given to content data based on the feature vector with reduced dimensions and the tag information given in advance generated by the dimension reduction feature generating means for each content data in the tagged learning data set. A classification model learning means for learning a classification model based on a posteriori probability estimation for classification;
Input data feature extraction means for extracting the feature vector from the input content data;
Based on the matrix learned by the dimension reduction space learning means, feature dimension reduction means for generating the feature vector with reduced dimensions from the feature vector extracted for the input content data;
Classification means for classifying tag information to be given to the input content data based on the feature vector generated by the feature dimension reduction means and the classification model learned by the classification model learning means;
Media data analysis device including A media data analysis method in a media data analysis apparatus for assigning tag information, which is information for explaining content data, to given content data,
Content data that is an element of a learning data set with a tag that is a set of content data to which the tag information is previously assigned by the learning data feature extracting unit, and no tag that is a set of content data to which no tag information is given From each of the content data that is an element of the learning data set, a feature vector that is a vector expressing the characteristics of the content data is extracted,
The feature vector extracted for each content data of the tagged learning data set and the pre-assigned tag information by the dimension reduction space learning means, and the content data extracted for each content data of the untagged learning data set. Learning a matrix representing a mapping for reducing the dimensionality of the feature vector based on the feature vector;
Based on the matrix learned by the dimension reduction space learning means by the dimension reduction feature generation means, the dimension is reduced from each of the feature vectors extracted for each content data of the tagged learning data set. Generate each feature vector,
Content data based on the feature vector reduced in dimension and the tag information given in advance generated by the dimension reduction feature generation unit for each content data of the tagged learning data set by the classification model learning unit Learn a classification model based on posterior probability estimation to classify tag information given to
The feature vector is extracted from the input content data by the input data feature extraction means,
Based on the matrix learned by the dimension reduction space learning means, the feature dimension reduction means generates the feature vector with reduced dimensions from the feature vector extracted for the input content data,
Media data for classifying tag information to be given to the input content data based on the feature vector generated by the feature dimension reduction unit and the classification model learned by the classification model learning unit by a classification unit analysis method. A program for giving tag information, which is information explaining the content data, to given content data,
Computer
Content data that is an element of a tagged learning data set that is a set of content data to which the tag information is assigned in advance, and content that is an element of an untagged learning data set that is a set of content data to which no tag information is given Data feature extraction means for learning for extracting a feature vector that is a vector expressing the characteristics of content data from each of the data;
Based on the feature vector extracted for each content data of the tagged learning data set and the pre-assigned tag information, and the feature vector extracted for each content data of the untagged learning data set. Dimensional reduction space learning means for learning a matrix representing a mapping for reducing the number of dimensions of the feature vector;
Dimensions for generating the feature vectors with reduced dimensions from each of the feature vectors extracted for each content data of the tagged learning data set based on the matrix learned by the dimension reduction space learning means. Reduction feature generation means,
Tag information to be given to content data based on the feature vector with reduced dimensions and the tag information given in advance generated by the dimension reduction feature generating means for each content data in the tagged learning data set. A classification model learning means for learning a classification model based on a posteriori probability estimation for classification;
Input data feature extraction means for extracting the feature vector from the input content data;
Based on the matrix learned by the dimension reduction space learning means, feature dimension reduction means for generating the feature vector with reduced dimensions from the feature vector extracted for the input content data, and the feature A program for functioning as a classification means for classifying tag information to be given to the input content data based on the feature vector generated by the dimension reduction means and the classification model learned by the classification model learning means .