CN104679902A - Information abstract extraction method in conjunction with cross-media fuse - Google Patents

Abstract

The invention proposes an information abstract extraction method combined with cross-media fusion. Firstly, classify the input multimedia data (text, image, audio, video, etc.) according to the data type; then homogeneous heterogeneous multimedia data and establish the text label of the data to obtain the same-dimensional image and text label; then The two-dimensional image data is clustered and the relevance test of the text label is carried out; several same-dimensional images are then classified and fused into one image; finally, a cross-media information summary is generated. Users can view the fusion image of each type of information through the information summary, and can quickly access the corresponding multimedia data.

Description

An Information Abstract Extraction Method Combined with Cross-media Fusion

technical field

The invention relates to an information abstract extraction method combined with cross-media fusion, belonging to the field of information extraction.

Background technique

We live in an information age, where massive amounts of information are increasing, and the Internet is adding a large amount of information every day, and the storage methods of information are becoming more and more diverse. Text, images, audio, and video are the basic forms of multimedia resources. Nowadays, various types of media data coexist, and the organization structure of media data is complex, but different types of media data express the same semantics from different aspects. In information extraction, it is necessary to cross from one media to another according to the various connections between media. kind of media. Therefore, how to cross the boundary between media and how to extract the potential correlation between multiple media has become a challenge for information extraction.

For big data where multiple media forms coexist, the existing methods are mainly realized through the feature recognition of the same media, and it is difficult to bridge the semantic gap between multimedia, such as the visual features of images and the auditory features of audio. The dimensions are different and the similarity between them cannot be directly measured. Therefore, the existing information extraction methods cannot provide users with intuitive thumbnails (or information summaries). How to classify and extract a large amount of mixed multimedia data becomes information extraction. It is one of the key technical problems that need to be solved urgently, and it is also a hot topic of research at present.

Existing mature text mining technology, image feature extraction algorithm, audio scene recognition, speech recognition, video scene segmentation, key frame extraction and other methods can extract semantic information of a single media. How to combine these algorithms to combine features of different dimensions Information extraction, forming an information extraction system for processing multimedia, we solve this problem through the medium of image, which is an intermediate dimension.

Contents of the invention

In view of the above problems, the present invention proposes an information summary extraction method combined with cross-media fusion. By adopting the method of synchronizing different-dimensional data into images, it solves the problem that it is difficult to cross the multimedia semantic gap. Through the image clustering method, the multimedia data is classified and extracted indirectly, and the cross-media information summary is generated.

The invention proposes an information abstract extraction method combined with cross-media fusion. Firstly, the input multimedia data (text, image, audio, video, etc.) are classified according to the data type; then the multi-dimensional multimedia data is synchronized and the text label of the data is established to obtain the same-dimensional image and text label; then the same-dimensional image and text label are obtained. The two-dimensional image data is clustered and the relevance test of the text label is carried out; several same-dimensional images are then classified and fused into one image; finally, a cross-media information summary is generated. Users can view the fusion image of each type of information through the information summary, and can quickly access the corresponding multimedia data.

The present invention proposes a method for extracting information summaries combined with cross-media fusion, comprising the following steps:

Step 1: Classify the input multimedia data (text, image, audio, video) into original text data by data type , the original image data , the raw audio data , the original video data ;

Step 2: Set the standard value of the image data dimension (image pixel), establish a same-dimensional image sample library with text labels, carry out the same-dimensional processing of different-dimensional multimedia data, and adopt corresponding processing methods according to different data types;

Step 3: For the processed same-dimensional image data, determine the threshold according to the accuracy required for clustering , perform clustering according to the image clustering algorithm, perform text label correlation test according to the text labels of each type of data, and cluster the data that does not meet the conditions again until the number of data that does not meet the conditions is less than the threshold ,Available same-dimensional image data the address of the index ;

Step 4: Fusion the clustered same-dimensional image data according to a fusion rule to obtain a fusion image of each type of same-dimensional image data ;

Step 5: Generate an information summary according to the fused image and index of each type of same-dimensional image data.

Compared with existing methods, the advantages of the present invention are:

1. Express the semantics of different-dimensional multimedia data with image data of the same dimension, crossing the boundaries between various media, and use relevant algorithms of image processing to process multimedia data;

2. The image clustering method is combined with the text label correlation test to ensure the accuracy of the classification and the strong correlation between the data.

Description of drawings

Fig. 1 is a flow chart of the present invention;

Fig. 2 is a flow chart of the same-dimensionalization method for different-dimensional data in the present invention;

Fig. 3 is a schematic diagram of the same-dimensional image data clustering and text label correlation test in the present invention.

Specific implementation method

Below in conjunction with accompanying drawing and specific embodiment the present invention is described in further detail:

Step 1: Classify the input multimedia data (text, image, audio, video) into original text data by data type , the original image data , the raw audio data , the original video data .

Step 2: Refer to Figure 2, set the standard value of the image data dimension (image pixel), establish a sample library of the same dimension image with text labels, perform the same dimension processing of different dimension multimedia data, and use corresponding methods according to different data types processing method;

The existing classification results are group of raw text data, group of raw image data, group of raw audio data, group of raw video data, the group raw text data Processing as same-dimensional image data ,Will Group raw image data Processing as same-dimensional image data ,Will Group raw audio data Processing as same-dimensional image data ,Will Group raw video data Processing as same-dimensional image data , the detailed steps are as follows;

1) Convert raw text data to Processing as same-dimensional image data process and related operations;

a) Preprocessing, using some text mining technology (such as text mining based on semantic understanding), the original text data The keywords of each group of text information paragraphs in the text are extracted as tags ;

b) will The group text data is mapped to the same-dimensional image data according to the tag keywords and sample library , where a set of text can correspond to multiple labels and same-dimensional image data, The corresponding sample image can be expressed as .

2) Convert the original image data to Processing as same-dimensional image data process and related operations;

a) Preprocessing raw image data , using related algorithms to enhance key features (such as removing background areas) to obtain processed images ;

b) for images , use some image scaling technology (such as bicubic interpolation and wavelet inverse interpolation) to scale to the same dimension image data (same dimension as the sample library);

c) Convert the same-dimensional image data Use a certain recognition method (such as image feature extraction algorithm based on visual information) to compare with the sample library to obtain the text label of the image, and the result is stored in .

3) Convert the raw audio data to Processing as same-dimensional image data process and related operations;

a) Preprocess raw audio data , use related algorithms to extract key features such as audio scene (such as audio scene recognition method based on probabilistic latent semantic analysis), language semantics (such as neural network-based speech recognition), and get the extracted text label ;

b) For the extracted text labels , the text label corresponds to the sample library, and the same-dimensional image data is obtained , where the same group of audio can correspond to multiple tags and image data of the same dimension, The corresponding multiple sample images can be expressed as .

4) Convert the raw video data to Processing as same-dimensional image data process and related operations;

a) Preprocessing raw video data , using a scene segmentation algorithm (such as a semantic-based video scene segmentation algorithm), for each video , after getting the segmented scene video clips ;

b) For each video segment of , using a certain key frame extraction method (such as multi-feature fusion key frame extraction based on clustering algorithm) to obtain the key frame image , the set of key frame images of each video is denoted as ;

c) For keyframe images , using related algorithms to enhance key features (such as removing background areas);

d) Use certain image scaling techniques (such as bicubic interpolation and wavelet inverse interpolation) to scale the processed image to the same dimension image data (same dimension as the sample library);

e) Convert the same-dimensional image data , using a certain recognition method to compare with the sample library to obtain the text label of the image, and the result is stored in .

Step 3: See Figure 3, for the processed same-dimensional image data, determine the threshold according to the accuracy required for clustering , perform clustering according to a certain image clustering algorithm (such as image clustering based on genetic algorithm), perform text label correlation test according to the text labels of each type of data, and re-cluster the data that does not meet the conditions until the conditions are not satisfied The number of data is less than the threshold , available index ,for same-dimensional image data address, the detailed steps are as follows:

1) For the processed same-dimensional image data, determine the threshold according to the accuracy required for clustering , The smaller the value, the greater the number of classifications and the more accurate the classification; otherwise, the fewer the number of classifications;

2) Perform clustering according to a certain image clustering algorithm (such as image clustering based on genetic algorithm), store the address of the clustered same-dimensional image, and extract the corresponding text label for the same type of clustered same-dimensional image , performing a text label correlation test between the text label and the image clustering result;

3) For clustered image data of the same dimension, if the number that does not meet the text label relevance test condition is greater than the threshold , then the data that does not meet the conditions will be removed from this category, and it will become unclustered same-dimensional image data again, and will be clustered again according to the same or different clustering methods until the number of data that does not meet the conditions is less than the threshold ;

4) Store the classification results in the form of addresses to get the index ,for same-dimensional image data the address of.

Step 4: Fusion the clustered same-dimensional image data according to a certain fusion rule (such as selecting one image with more targets), so as to obtain the fusion image of each type of same-dimensional image data ;

by index take out same-dimensional image data , according to a certain fusion rule, fusion is performed to obtain the fusion image of each type of same-dimensional image data .

Step 5: Generate an information summary according to the fusion image and index of each type of same-dimensional image data;

The resulting fused image will be and the index Generating a summary of information , the user can view the fused image and access the corresponding multimedia data.

Claims (8)

1. combine the informative abstract extracting method across Media Convergence, it is characterized in that, first the multi-medium data (word, image, audio frequency, video etc.) of input is classified by data type; Again different dimension multi-medium data is set up the text label of data with dimensionization, obtain dimension image and text label together; Then by same dimensional data image cluster and carry out text label relevance inspection; Several same dimension images of sub-category fusion are a sub-picture again; Finally generate and make a summary across media information; Described method at least comprises the following steps:

Step one: be urtext data by data type classifications by (word, image, audio frequency, video) in the multi-medium data of input , raw image data , original audio data , original video data ;

Step 2: arrange view data dimension (image pixel) standard value, sets up the same Wei Tuxiangyangbenku with text label, carries out different dimension multi-medium data with dimensionization process, adopts corresponding disposal route according to the difference of data type;

Step 3: to processed same dimensional data image, the accuracy definite threshold required for cluster , carry out cluster according to image clustering algorithm, carry out the inspection of text label relevance according to the text label of every class data, by the data cluster again do not satisfied condition, until the data bulk do not satisfied condition is less than threshold value , can obtain roughly the same dimensional data image address, i.e. index ;

Step 4: to the same dimensional data image of cluster, according to a kind of fusion rule, merge, thus the fused images obtaining each roughly the same dimensional data image ;

Step 5: according to each the roughly the same fused images of dimensional data image and index, information generated is made a summary.

2. a kind of combination according to claim 1 is across the informative abstract extracting method of Media Convergence, to it is characterized in that in step 2 and by urtext data be treated to same dimensional data image process and associative operation, at least further comprising the steps of:

1) pre-service, utilizes Text Mining Technology, by urtext data in often organize text message paragraph keyword extraction be label ;

2) will group text data corresponds to same dimensional data image according to label keyword and Sample Storehouse , wherein, one group of text may correspond to multiple label and same dimensional data image, corresponding sample image can be expressed as .

3. a kind of combination according to claim 1 is across the informative abstract extracting method of Media Convergence, to it is characterized in that in step 2 and by raw image data be treated to same dimensional data image process and associative operation, at least further comprising the steps of:

1) pre-service raw image data , utilize related algorithm to strengthen key feature, obtain the image after processing ;

2) for image , utilize image scaling techniques to be scaled same dimensional data image (with Sample Storehouse with tieing up);

3) by same dimensional data image adopt image-recognizing method and Sample Storehouse comparison, obtain the text label of image, result is deposited in .

4. a kind of combination according to claim 1 is across the informative abstract extracting method of Media Convergence, it is characterized in that original audio data in step 2 be treated to same dimensional data image process and associative operation, at least further comprising the steps of:

1) pre-service original audio data , utilize related algorithm to extract audio scene, the key features such as language semantic, obtain the text label extracted ;

2) for the text label extracted , text label is corresponding with Sample Storehouse, obtains same dimensional data image , wherein, may correspond to multiple label and same dimensional data image with group audio frequency, corresponding multiple sample images can be expressed as .

5. a kind of combination according to claim 1 is across the informative abstract extracting method of Media Convergence, it is characterized in that original video data in step 2 be treated to same dimensional data image process and associative operation, at least further comprising the steps of:

1) pre-service original video data , utilize Algorithm of Scene, for each video , after obtaining split sence individual video segment ;

2) for each video segment , adopt extraction method of key frame, obtain key frame images , the set of the key frame images of each video is designated as ;

3) for key frame images , utilize related algorithm to strengthen key feature;

4) same dimensional data image is scaled to processed imagery exploitation image scaling techniques (with Sample Storehouse with tieing up);

5) by same dimensional data image , adopt image-recognizing method and Sample Storehouse comparison, obtain the text label of image, result is deposited in .

6. a kind of combination according to claim 1 is across the informative abstract extracting method of Media Convergence, it is characterized in that to same dimension image clustering and the process setting up index in step 3, at least further comprising the steps of:

1) to processed same dimensional data image, the accuracy definite threshold required for cluster , less, classification quantity is more, classifies more accurate, otherwise classification quantity is fewer;

2) carry out cluster according to image clustering algorithm, store the same dimension image address of cluster, for the same class of cluster with tieing up image, extract the text label of its correspondence, the relevance of carrying out text label and image clustering result is checked;

3) for the same dimensional data image of cluster, if the quantity not meeting test condition is greater than threshold value , then the data do not satisfied condition are rejected this class, again become the same dimensional data image of non-cluster, and according to identical or different method cluster again, until the data bulk do not satisfied condition is less than threshold value ;

4) classification results is stored with the form of address, obtain index , for roughly the same dimensional data image address.

7. a kind of combination according to claim 1 is across the informative abstract extracting method of Media Convergence, it is characterized in that in step 4, sub-category fusion is the process of a width with dimensional data image, at least further comprising the steps of:

1) index is pressed successively take out roughly the same dimensional data image , according to a kind of fusion rule, merge, thus obtain the fused images of each roughly the same dimensional data image .

8. a kind of combination according to claim 1 is across the informative abstract extracting method of Media Convergence, it is characterized in that according to the fusion of each category information same dimension image and index in step 5, the process of information generated summary, at least further comprising the steps of:

1) fused images will obtained and index information generated is made a summary .