CN114580557A - Method and device for determining the similarity of documents based on semantic analysis - Google Patents

Abstract

The invention discloses a semantic analysis-based document similarity determination method and device, wherein the method comprises the following steps: dividing each document to be compared into a plurality of parts; performing semantic analysis on each part to obtain a semantic analysis result of each part; determining a weight value of each part of each document to be compared according to the semantic analysis result of each part; obtaining a weighted average result of each document to be compared according to the weight value of each part of each document to be compared; and determining the similarity between the documents to be compared according to the weighted average result of each document to be compared. The method and the device can accurately determine the weights of different parts of the literature based on semantic analysis, and further accurately determine the similarity of the literature.

Description

Method and device for determining the similarity of documents based on semantic analysis

technical field

The invention relates to the technical field of artificial intelligence, and in particular, to a method and device for determining the similarity of documents based on semantic analysis.

Background technique

This section is intended to provide a background or context to the embodiments of the invention recited in the claims. The descriptions herein are not admitted to be prior art by inclusion in this section.

At present, when determining the similarity of documents in the prior art, according to manual experience, different weights are preset for different parts of the documents, and finally, according to the fixed weights set artificially, the result obtained by the weighted summation of the similarity of each part of the content is determined as the document. similarity. The existing methods for determining the similarity of documents set the weights based on experience, and the weight setting is inaccurate, which leads to the problem of inaccurate determination of the similarity of documents.

SUMMARY OF THE INVENTION

An embodiment of the present invention provides a method for determining the similarity of documents based on semantic analysis, which is used to accurately determine the weights of different parts of documents based on the semantic analysis, and then accurately determine the similarity of documents. The method includes:

Divide each document to be compared into multiple sections;

Perform semantic analysis on each part to obtain the semantic analysis result of each part;

According to the semantic analysis result of each part, determine the weight value of each part of each document to be compared;

According to the weight value of each part of each document to be compared, the weighted average result of each document to be compared is obtained;

According to the weighted average result of each document to be compared, the similarity between documents to be compared is determined.

The embodiment of the present invention also provides a device for determining the similarity of documents based on semantic analysis, which is used to accurately determine the weights of different parts of the documents based on the semantic analysis, and then accurately determine the similarity of documents. The device includes:

A division unit, used to divide each document to be compared into a plurality of parts;

The semantic analysis unit is used to perform semantic analysis on each part to obtain the semantic analysis result of each part;

a weight value determination unit, used for determining the weight value of each part of each document to be compared according to the semantic analysis result of each part;

a processing unit, configured to obtain a weighted average result of each document to be compared according to the weight value of each part of each document to be compared;

The similarity determination unit is configured to determine the similarity between the documents to be compared according to the weighted average result of each document to be compared.

An embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored in the memory and running on the processor, when the processor executes the computer program, the above-mentioned documents similar to those based on semantic analysis are implemented degree determination method.

An embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the above-mentioned method for determining the similarity of documents based on semantic analysis is implemented.

An embodiment of the present invention further provides a computer program product, the computer program product includes a computer program, and when the computer program is executed by a processor, the above-mentioned method for determining the similarity of documents based on semantic analysis is implemented.

In the embodiment of the present invention, the solution for determining the similarity of documents based on semantic analysis is different from the prior art, which presets different fixed weights for different parts of the documents based on experience, and then determines the similarity of documents. Compared with the technical solutions for the problem of inaccurate similarity determination, by: dividing each document to be compared into multiple parts; performing semantic analysis on each part to obtain the semantic analysis result of each part; according to the semantic analysis result of each part, Determine the weight value of each part of each document to be compared; obtain the weighted average result of each document to be compared according to the weight value of each part of each document to be compared; By comparing the similarity between documents, the weight of different parts of documents can be accurately determined based on semantic analysis, and then the similarity of documents can be accurately determined.

Description of drawings

In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative efforts. In the attached image:

1 is a schematic flowchart of a method for determining the similarity of documents based on semantic analysis in an embodiment of the present invention;

2 is a schematic flowchart of performing semantic analysis on each part to obtain a semantic analysis result of each part in an embodiment of the present invention;

3 is a schematic flowchart of performing semantic analysis on each part to obtain a semantic analysis result of each part in another embodiment of the present invention;

4 is a schematic flowchart of a document preprocessing process in an embodiment of the present invention;

5 is a schematic structural diagram of a device for determining the similarity of documents based on semantic analysis in an embodiment of the present invention;

6 is a schematic structural diagram of a semantic analysis unit in an embodiment of the present invention;

7 is a schematic structural diagram of a semantic analysis unit in another embodiment of the present invention;

FIG. 8 is a schematic diagram of a training process of Chinese word segmentation in a professional field according to an embodiment of the present invention.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention more clearly understood, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but not to limit the present invention.

1 is a schematic flowchart of a method for determining the similarity of documents based on semantic analysis in an embodiment of the present invention. As shown in FIG. 1 , the method includes the following steps:

Step 101: Divide each document to be compared into multiple parts;

Step 102: perform semantic analysis on each part to obtain a semantic analysis result of each part;

Step 103: Determine the weight value of each part of each document to be compared according to the semantic analysis result of each part;

Step 104: According to the weight value of each part of each document to be compared, obtain the weighted average result of each document to be compared;

Step 105: Determine the similarity between the documents to be compared according to the weighted average result of each document to be compared.

The method for determining the similarity of documents based on semantic analysis provided by the embodiment of the present invention, when working: divides each document to be compared into multiple parts (which may be called sub-documents); performs semantic analysis on each part to obtain the semantic analysis of each part Result; according to the semantic analysis result of each part, determine the weight value of each part of each document to be compared; according to the weight value of each part of each document to be compared, obtain the weighted average result of each document to be compared; The weighted average result of the documents to be compared is used to determine the similarity between the documents to be compared.

Compared with the technical solution in the prior art that different fixed weights are preset for different parts of the document according to experience, and then the document similarity is determined, and the weight setting is inaccurate, which leads to the problem of inaccurate determination of the document similarity. The method for determining the similarity of documents based on semantic analysis provided by the embodiment can accurately determine the weights of different parts of documents based on the semantic analysis, and then accurately determine the similarity of documents. The method for determining the similarity of documents based on semantic analysis is described in detail below.

1. First, the above step 101 is introduced.

During specific implementation, the documents in the embodiments of the present invention may be patent documents, trademark documents, non-patent documents, and the like. Taking patent documents as an example, each document to be compared is divided into multiple parts, such as the title of the invention, the abstract, the claims, and the description.

2. Next, the above step 102 is introduced.

In one embodiment, as shown in FIG. 2 , performing semantic analysis on each part to obtain a semantic analysis result of each part may include the following steps:

Step 1021: Perform word segmentation processing on each part to obtain multiple keywords corresponding to each part;

Step 1022: According to the multiple keywords corresponding to each part and the preset document feature extraction strategy, extract multiple types of key features from each part to form a feature set corresponding to each part;

Step 1024: According to the feature set corresponding to each part, perform word-level, syntactic-level and chapter-level semantic analysis on each part to obtain a semantic analysis result of each part.

During the specific implementation, in the above step 1021, document word segmentation processing: according to the thesaurus, the document document is subjected to word segmentation processing, and the function words lacking actual meaning, the rare low frequency words and the frequently used high frequency words are deleted, and finally each word is obtained. Some of the corresponding keywords.

During specific implementation, a pre-trained word segmentation model can be used for word segmentation processing. In order to better understand how the word segmentation process of the present invention is implemented, the word segmentation model is introduced below.

1. Evaluation metrics.

Chinese word segmentation adopts the evaluation indicators of Precision and Recall, among which:

Precision=Number of correctly segmented words/Total number of segmented words;

Recall = the number of correctly segmented words/the total number of words that should be segmented;

Comprehensive performance index F-measure;

Fβ=(β2+1)×Precision×Recall/(β2×Precision+Recall);

β is the weight factor. If the precision rate and the recall rate are treated equally, and β=1, the most commonly used F1-measure is obtained;

F1=2×Precisiton×Recall/(Precision+Recall).

The following takes the word segmentation model as a Chinese word segmentation model as an example to introduce.

2. Chinese word segmentation model training method.

Commonly used word segmentation methods can be divided into two categories: dictionary-based word segmentation and statistical-based word segmentation. The statistical-based word segmentation method has become the mainstream word segmentation method in recent years because it has greatly improved compared with the dictionary-based word segmentation in terms of ambiguity segmentation and unregistered word recognition. Commonly used statistical models for word segmentation include Hidden Markov Model, Conditional Random Field Model, Maximum Entropy Model, and Neural Network Model. However, when the test corpus is inconsistent with the training corpus, the accuracy of word segmentation and the performance of unregistered word recognition will drop significantly. Therefore, when using the statistical-based word segmentation method to segment the professional domain text, it is necessary to make annotated training corpus for the corresponding domain. However, labeling training corpora in professional fields will consume a lot of manpower and material resources, and the number of professional fields that have completed labeling work at this stage is rare.

Domain terminology embodies and loads the core knowledge of a subject area, and professional domain dictionary is a collection of professional domain terminology. The methods of constructing specialized domain dictionaries can be divided into: constructing domain dictionaries using dictionary resources, constructing domain dictionaries using corpus and statistical methods, and constructing domain dictionaries using encyclopedia mining. The inventor proposes a professional domain dictionary construction method combining dictionary resources and encyclopedia, using encyclopedia to obtain new terms, and using existing dictionary resources to improve the completeness.

The training process of Chinese word segmentation in the professional field can be shown in Figure 8:

(1) Construct a domain dictionary by mining encyclopedia entries and combining existing dictionary resources.

(2) Take the word segmentation result in the basic natural language processing model as the primary word segmentation result, use the domain dictionary to perform reverse maximum matching segmentation on the result, and combine the ambiguity resolution rules to obtain the secondary word segmentation result.

After using the basic natural language processing model tokenizer to obtain the word segmentation results, the domain dictionary is used to perform reverse maximum matching on the word segmentation results, which can improve the domain self-applicability of word segmentation. The reverse maximum matching algorithm extracts the string of word segmentation results from right to left, and matches it in the domain dictionary. If the match is successful, the string is divided into words. For example: "replace rock wool composite board with benzene board composite board." Use the basic natural language processing model tokenizer to segment words, and get: "rock/cotton/composite/board/replace/for/benzene/board/composite/board/." The domain dictionary includes "rock wool", "benzene board", "composite board". Perform reverse maximum matching, and the adjusted word segmentation result is: "rock wool/composite board/replacement/for/benzene board/composite board/."

Directly using the domain dictionary to perform reverse maximum matching adjustment on a word segmentation result often faces ambiguity problems. For example, "after the expiration of the term", the result of a word segmentation is: "after/work/expiration/after". Both "duration" and "expiration" are assumed to be in the domain dictionary, with two possible outcomes: "at/duration/expiration/after" and "at/duration/expiration/after". Directly use the reverse maximum matching method to obtain the former result, however, the correct segmentation result should be the latter. Therefore, it is necessary to design a disambiguation algorithm to improve the accuracy of word segmentation.

Design principle: When the maximum reverse matching algorithm adjusts the initial word segmentation result, the following situations will occur: keep the initial word segmentation result unchanged; merge the words in the initial word segmentation result without affecting other words; the words in the initial word segmentation result are re-segmented , form new words. The disambiguation algorithm we propose is mainly aimed at the third case. The disambiguation algorithm is used to decide whether to merge the strings currently matched to the domain dictionary into new words. The disambiguation algorithm proposed by us mainly considers the number of words before and after adjustment, the number of single-character words, and the position characteristics of words in words. If the number of words after adjustment increases, no adjustment will be made. If the number of words after adjustment increases, no adjustment will be made. The positional features of words refer to 6 word lists about the first word, middle word and last word, which are the first word list (L1) with high frequency of single-word segmentation and the very low frequency of segmentation into single-word words. The first character list (L2), the Chinese character list with a high frequency of combining with the first character (L3), the high frequency of combining with the last character to form a Chinese character list (L4), and the frequency of splitting into single-character words. High end word list (L5) and end word list (L6) with very low frequency of segmentation into single-word words.

3. Chinese word segmentation model training process.

The first word segmentation result S: a1a2a3/a4/…/aM-1aM, W0 represents the word currently matched in the domain dictionary, HD(W0) represents the first word of W0, TL(W0) represents the last word of W0, LEN( W0) represents the length of W0, and minSub(S, W0) represents the smallest subsequence in S that includes W0, and the subsequence starts and ends with the segmentation interval. combine(S, W0) represents the sequence after combining W0 in S, LMatch(ai) represents the starting point of ai, and the reverse maximum matching of the word is obtained in the domain dictionary, and the left segmentation point of the word does not fall on the in existing words. new(S, W0) represents the new sequence set with the number of words greater than or equal to 2 after merging W0 in S. Assuming that the result of a word segmentation is a1a2a3/a4 a5/a6a7a8, and W0 is a5a6, then HD(W0) is a5, TL(W0) is a6, minSub(S,W0) is a4a5/a6a7a8, combine(minSub(S,W0) ,W0) is a4/a5a6/a7a8, and the leftmost end of combine(minSub(S,W0),W0) is the single word a4. If a2a3a4 is in the domain lexicon, LMatch(a4) is empty because the left split point falls in the existing word a1a2a3. new(S, W0)={a7a8}.

When LEN(W0) = 3, when the matching word length is 3 characters, priority is given to whether the left and right sides generate single characters and word position features, secondly, the case where minSub(S, W0) is composed of single characters and 2 characters is considered, and finally single characters are considered The specific rules for changing the number of words and changing the number of words are as follows.

Rule 1: If the leftmost end of combine(minSub(S,W0),W0) is a single word, denoted as aL, if LMatch(aL) is empty, and the rightmost end of combine(minSub(S,W0),W0) is not a single word, then , if aL ∈ L1, and the number of words in combine(minSub(S, W0), W0) is less than or equal to minSub(S, W0), combine W0.

Rule 2: If the rightmost end of combine(minSub(S,W0),W0) is a single word (aR) and the leftmost end is not a single word, or the leftmost end of combine(minSub(S,W0),W0) is a single word (aL), but LMatch (aL) is not empty, then if aR is in the L5 word list, and the number of words in combine(minSub(S, W0), W0) is less than or equal to minSub(S, W0), combine W0.

Rule 3: minSub(S, W0) is A/BC, LMatch(A) is not empty, W0 is not merged, and the next match starts from A.

Rule 4: If the number of single-word words in combine(minSub(S,W0),W0) is 0, and the number of words is less than or equal to minSub(S,W0), at the same time, the sequence in new(S,W0) is in the NLPIR dictionary or In the domain dictionary, W0 is merged. If the leftmost end of combine(minSub(S, W0), W0) is a single-word word aL, and LMatch(aL) is not empty, then aL is not counted as a single-word word.

Rule 5: If W0 is AB, minSub(S, W0) is A/B*, and * represents a string with a length greater than 2, then if A is in L1, do not merge W0, otherwise, merge W0.

Rule 6: If W0 is AB, minSub(S, W0) is *A/B, and * represents a string with a length greater than 2, then if B is in L5, do not merge W0, otherwise, merge W0.

Rule 7: Except for rules 2 to 6, W0 is not merged.

The above rules are executed in order. In the actual effect verification, compared with the direct use of the tokenizer in the basic natural language processing model, the Chinese word segmentation model we trained has greatly improved in accuracy, recall, and F value.

During specific implementation, in the above step 1022, feature extraction (extraction):

In one embodiment, the preset document feature extraction strategy may include: according to the frequency of the keyword appearing in the document, the inverse document frequency of the keyword, the part of speech of the keyword, whether the keyword is a professional word, the occurrence of the keyword The position in the literature, the text-rank value of the keyword, the information entropy value of the keyword, the word vector of the keyword and the overall deviation value, the length of the keyword, the keyword as a component of the sentence, whether the keyword is further divided into sub-sections One or any combination of keywords, the length of the first occurrence and the last occurrence of keywords in the document, and the deviation of the keyword distribution, are used for document feature extraction.

In specific implementation, the above document feature extraction strategy can improve the accuracy of feature extraction, thereby improving the accuracy of semantic analysis, thereby improving the accuracy of determining the weight value of each part of each document to be compared, and finally improving the accuracy of document similarity determination.

In one embodiment, when the keywords can be further divided into sub-keywords, the preset document feature extraction strategy may further include: according to the word frequency of the sub-keywords-inverse document frequency, the part-of-speech of the sub-keywords, Whether the sub-keyword is one of the specialized words or any combination is used for feature extraction of documents.

During specific implementation, the above-mentioned document feature extraction strategy considering sub-keywords can further improve the accuracy of feature extraction, thereby further improving the accuracy of semantic analysis, and further improving the accuracy of determining the weight value of each part of each document to be compared. , and finally further improve the accuracy of document similarity determination.

To sum up, in order to facilitate understanding of how the present invention is implemented, the document feature extraction strategy is further described below in the form of Table 1 below.

In specific implementation, after analyzing the data features and business scenarios of patent documents, we propose a specific strategy for extracting features from patent documents using natural language processing technology, as shown in Table 1 below:

Table 1

As shown in Table 1 above, compared with the prior art scheme in which fixed weights are set for different parts of documents to determine the similarity of documents based on experience, the embodiment of the present invention is equivalent to adjusting the weights based on semantic analysis, thereby obtaining a more accurate The weight value is used to improve the accuracy of document similarity determination.

In one embodiment, the plurality of types of key characteristics may include: document static characteristics, characteristics of documents associated with the query, and characteristics of the query.

In specific implementation, features can be the source of nutrients for algorithms and models. The quality of feature selection is directly related to the effect of the model trained by the algorithm. Different from traditional text classification, the output of MLR in the patent field is the ranking of the document collection given the query (query or retrieval feature), which not only considers the characteristics of the document itself, but also the characteristics of the relationship between the query and the document. To sum up, MLR in the patent field needs to consider the characteristics of three aspects (document static characteristics, characteristics associated with documents and queries, and characteristics of queries):

The feature selection of patent information refers to the purpose of grasping the development status of a certain technology by summarizing, deducting, analyzing, synthesizing, abstracting and generalizing the internal features of patent documents, that is, the content of patent technology. Specifically, according to the technical content provided by the patent documents, such as the technical subject, patent country, patent inventor, patent assignee, patent classification number, patent application date, patent grant date and patent citation documents, extensive information collection is carried out, and the The collected content is read and excerpted, and on this basis, further research activities such as classification, comparison and analysis of the information are carried out to form an organic information collection. And then focus on those representative, key and typical patent documents, and finally find out the internal and even potential relationship between patent information, so as to form a more complete understanding.

1. The static features of the document itself (document static features), including the text features of the document, such as word vectors with weights, TF and IDF in different domains of the document (title, bibliographic item abstract, claims, patent specification, patent text, etc.) , BM25, and other language model scores, as well as document quality scores, patent claims, and other importance scores. Regarding the quality score of documents, the search has different calculation indicators according to different patent classifications. For example, the calculation of the quality score of patent documents in the mechanical field should not only consider the text richness of the patent itself, but also consider various related content of the patent. Such as the change of part of speech, the position of words, the syntactic structure of mechanical domain, etc.

2. Features associated with documents and queries (features associated with documents and queries), such as the TD-IDF score and BM25 score of the document corresponding to the query.

3. The characteristics of the query itself (the characteristics of the query), such as the policy text characteristics, the weighted professional word vector in the patent field, the length of the query, the classification described by the query, the sum/avg/min/max/median score of the BM25 of the query, The popularity of query last month, etc.

In the feature engineering of query and document, in addition to lexical analysis, it is also necessary to analyze and extract from the semantics, that is, the concept, which is "really expressed" by the "explained" lexical. For example, polysemy, synonyms and synonyms, the same word in different contexts expresses different meanings, and different words in different contexts may also express the same meaning. LSA (Latent Semantic Analysis) is a well-known technique for dealing with such problems. Its main idea is to map a high-dimensional vector space to a low-dimensional latent semantic space or concept space, that is, dimensionality reduction. The specific method is to do singular value decomposition (SVD) of the term document matrix. In singular value decomposition, C is a matrix with documents as rows and terms as columns (assuming M x N), and the elements are term frequency-inverse document frequency values. C is decomposed into 3 small matrices and multiplied; each column of U represents a topic, and each non-zero element in it represents the correlation between a topic and an article, and the larger the value, the more relevant; V represents the correlation between keyword and all terms ∑ represents the correlation between the article topic and the keyword.

In one embodiment, as shown in FIG. 3 , the above-mentioned method for determining the similarity of documents based on semantic analysis may further include step 1023 : using principal component analysis, linear discriminant analysis and mutual information method, for each part of the corresponding feature set Screen and combine features to obtain feature sets corresponding to each part after feature dimensionality reduction processing;

According to the feature set corresponding to each part, perform word-level, syntactic-level and chapter-level semantic analysis on each part, and obtain the semantic analysis result of each part, which may include: according to the feature set corresponding to each part after feature dimension reduction processing, Each part is semantically analyzed at the word level, syntax level and text level, and the semantic analysis result of each part is obtained.

During specific implementation, the number of words that constitute a text is generally very large, so the dimension of the vector space representing the text is also quite large, which can reach thousands of dimensions, so dimensionality reduction processing is necessary. Dimensionality is generally reduced by the method of feature extraction, and the feature indicators representing vocabulary may include: document frequency, information acquisition, mutual information, square root fitting test, term strength, etc. By calculating any of the above indicators of the vocabulary, and then sorting from large to small to select a specified number of words or words whose index value is greater than a specified threshold to form a feature set. The dimensionality reduction process is further introduced below.

In specific implementation, after the patent documents are converted into feature sets, that is, in step 1023 after step 1022, many algorithms can directly enter the retrieval and sorting stage, but because the data of patent documents is too large, the characteristics of each patent document are And there are many, the calculation speed of the algorithm is very important, otherwise the algorithm may look beautiful, but it has no practical effect. In order to further speed up the calculation, on the basis of the feature set, principal component analysis, linear discriminant analysis, mutual information and other methods are used to screen and combine features to obtain new features and achieve the purpose of feature dimensionality reduction.

In specific implementation, dimensionality reduction is a method of preprocessing high-dimensional feature data. Dimensionality reduction is to retain some of the most important features of high-dimensional data, remove noise and unimportant features, so as to achieve the purpose of improving data processing speed. In actual production and application, dimensionality reduction can save us a lot of time and cost within a certain range of information loss. Dimensionality reduction has also become a very widely used data preprocessing method.

When implemented, dimensionality reduction has the following advantages:

1. Make the dataset easier to use.

2. Reduce the computational cost of the algorithm.

3. Remove noise.

4. Make the results easy to understand.

In the above step 1024, the natural language processing engine is used to perform semantic analysis at the word level, syntax level, and text level on the patent document. The word level analysis includes Chinese word segmentation, named entity recognition, part-of-speech tagging, synonym analysis, and word vector analysis. , n-gram analysis, word granularity analysis, stop word analysis, etc.; patent document syntactic level analysis includes dependency grammar analysis, language model, short string analysis, etc.; patent document chapter level analysis includes patent document label extraction, topic model, text aggregation class etc.

During specific implementation, in the above step 1024, feature evaluation weight: the calculation method of the feature weight is mainly to use the word frequency-inverse document frequency formula. According to different mining purposes, there are currently a variety of word frequency-inverse document frequency formula construction methods. It can be seen from the above content that the construction of the text vector space is carried out completely according to the laws of probability and statistics, and does not consider the relationship between words. Another key step in the text mining process is to calculate the similarity distance between text vectors. For any two vectors X=(x1,x2,...,xn) and X'=(x'1,x'2,..., x'n), there are three most commonly used distance measures: Euclidean distance, cosine distance and inner product. The above three distance measures also do not involve the analysis of the relationship between words in vectors. Through the above analysis, it can be concluded that both the text vector construction process and the similarity distance measurement process are designed according to the principle of probability and statistics, and do not consider the semantic relationship of features. In the use of natural language, synonyms and related words (words that are often used in combination according to the context) often appear in the text in order to express the needs. For example, in IT technology, "computer" and "computer" are synonymous; There is a very high chance that the "police" and the "case" will appear at the same time. These synonyms and related words also appear in a large number of text feature vectors. On the one hand, the dimension of the text feature vector is increased, and on the other hand, the expression accuracy of the text feature vector to the document is reduced. Although text feature extraction can reduce the dimension of feature vectors by pre-set thresholds, it is not based on the premise of ensuring semantic accuracy, so it is often counterproductive. Although it is also possible to use synonym dictionaries and entailment dictionaries to reduce synonyms and associated words in the process of word segmentation, it also brings problems of dictionary maintenance and updating.

Through the above analysis, it can be seen that it is very important to associate synonyms and related words with a given word, whether it is for user applications or for text semantic mining. The function of the retrieval association model is mainly to solve this problem, and the retrieval association model will be introduced below.

3. Next, in order to facilitate understanding, the above steps 103 to 105 are introduced together.

During specific implementation, the weight value of each part of each document to be compared is determined according to the semantic analysis result of each part; the weighted average result of each document to be compared is obtained according to the weight value of each part of each document to be compared; The weighted average result of each document to be compared determines the similarity between documents to be compared.

4. Next, further preferred steps are introduced.

The embodiment of the present invention can be used for semantic retrieval, and each document in the document database to be retrieved is subjected to document preprocessing. Taking patent documents as an example, patent document document preprocessing is performed by processing, extracting, encoding, converting, and returning documents. It is converted into semi-structured data containing plain text, and these semi-structured data will be used as input for intelligent patent retrieval to be retrieved from the same pre-established semi-structured data. The analyzed literature similarity determination methods are compared and sorted to obtain intelligent search results, which can also be called semantic search results, with high retrieval accuracy. Documents and their processing results are delivered through the Kafka distributed message queue. The whole process is shown in Figure 4:

The main format of patent document data is XML. First, the required data needs to be extracted and spliced from the two XML data tables.

XML files are mostly used for the description of information, so after obtaining an XML document, extracting the corresponding information according to the elements in the XML is the parsing of XML. There are two ways of XML parsing, one is DOM parsing, and the other is SAX parsing. The embodiment of the present invention adopts the DOM parsing way to process XML.

An XML parser based on DOM parsing converts it into a collection of object models and stores information in a tree data structure. Through the DOM interface, the application can access any part of the data in the XML document at any time, so this way of using the DOM interface to access is also called random access.

This method is also flawed, because the DOM analyzer converts the entire XML file into a tree and stores it in memory. When the file structure is large or the data is complex, this method has higher memory requirements, and for complex structures Tree traversal is also a very time-consuming operation. However, the tree structure used by DOM is consistent with the way XML stores information, and its random access can also be used, so the DOM interface still has extensive use value.

There are four core operation interfaces in DOM parsing:

1.Document: This interface represents the entire XML document, represented as the root of the entire DOM, which is the entry of the tree, through which the content of all elements in XML can be accessed. The commonly used methods are as follows:

public NodeList getElementByTagName(String tagname);

Get the NodeList of the specified node name;

Public Element createElement(String tagName) throws DOMException;

Create a node with the specified name;

Public Text createTextNode(String data) throws DOMException;

Create a text content node;

Element createElement(String tagName) throws DOMException;

create a node element;

Public Attr createAttribute(String name) throws DOMException;

Create a property:

2.Node: This interface plays a pivotal role in the entire DOM tree. The core interfaces of DOM operations are inherited from Node (Document, Element, Attr). In the DOM tree, each Node interface represents a DOM tree node.

Common methods of Node interface:

Node appendChilid(Node newChild) throws DOMException;

Add the next new node under the current node;

Public NodeList getChildNodes();

Get all child nodes under this node;

Public Node getFirstChild();

Get the first child node under the node;

Public Node getLastChild();

Get the last child node under this node;

Public boolean hasChildNodes();

Determine whether there are other nodes;

String getNodeValue() throws DOMException;

Get node content:

3.NodeList: This interface represents a collection of points, generally used for a set of nodes in an ordered relationship. Common methods of NodeList:

Public int getLength();

Get the number of nodes in NodeList;

Public Node item(int index);

Get the node object according to the index;

4. NamedNodeMap: This interface represents a one-to-one relationship between a group of nodes and their unique names, and is mainly used for the representation of node attributes.

In addition to the above four core interfaces, if a program needs to perform DOM parsing operations, it needs to follow the steps below:

1. Create a DocumentBuilderFactor to obtain a DocumentBuilder object:

DocumentBuilderFactory factory=DocumentBuilderFactory.newInstance();

2. Create DocumentBuilder:

DocumentBuilder builder=factory.newDocumentBuilder();

3. Create a Document object and get the entry of the tree:

Document doc=builder.parse("relative path or absolute path of XML file");

4. Create NodeList:

NodeList n1=doc.getElementByTagName("Read Node").

The following will introduce the intelligent retrieval of patent documents by using the method for determining the similarity of documents based on semantic analysis provided by the embodiments of the present invention.

1. Retrieval recall algorithm.

The most important thing in the retrieval system is sorting, which is divided into primary sorting (ie, retrieval recall) and learning sorting based on business understanding, NLP, and machine learning. Retrieval and recall mainly use algorithms such as word frequency-inverse document frequency and BM25.

The purpose of retrieval recall is to use a relatively lightweight algorithm to quickly obtain candidates, so as to facilitate subsequent sorting using complex sorting algorithms, optimize the search efficiency of the entire process, and avoid unnecessary operations. The candidate item is larger than the target item we are ultimately looking for. For example, our goal is to find the most likely 50 patent documents, then the candidates can be set to 500.

(1) Retrieval based on word frequency-inverse document frequency

Term Frequency-Inverse Document Frequency is a commonly used weighting technique for information retrieval and text mining. Term Frequency - Inverse Document Frequency is a statistical method used to assess the importance of a word to a document set or one of the documents in a corpus. In a given document, term frequency (tf) refers to the frequency with which a given word appears in the document. This number is normalized to the term count to prevent it from skewing towards long files. For a word ti in a particular document, its importance can be expressed as:

In the above formula, ni and j are the occurrences of the word in the file dj, and the denominator is the sum of the occurrences of all words in the file dj.

Inverse document frequency (idf) is a measure of the universal importance of a word. The idf of a particular word can be obtained by dividing the total number of documents by the number of documents containing the word, and then taking the base 10 logarithm of the obtained quotient to get:

in:

|D|: The total number of documents in the corpus;

|{j:t _i ∈d _j }|: The number of documents containing the word ti (ie the number of documents ni, j != 0) If the word is not in the data, the denominator will be zero, so 1+| is generally used {j:t _i ∈d _j }|;

Then tfidf i, j = tfi, j × idfi;

The high word frequency within a particular document, and the low document frequency of that word in the entire document collection, can result in a high weighted word frequency - inverse document frequency. Therefore, term frequency-inverse document frequency tends to filter out common terms and keep important terms.

(2) Recall based on BM25 retrieval

BM25 is a common formula for relevance scoring, which is equivalent to extending tfidf. The main purpose is to calculate the relevance of all words and documents in a query, and then accumulate the scores. The correlation score is mainly affected by tf/idf. The original formula of BM25 is as follows:

where ri is the number of related documents containing term i, ni is the number of documents containing term i, N is the number of all documents in the entire document dataset, and R is the number of documents related to this query, since in the typical case, There is no relevant information, that is, ri and R are both 0, and in normal queries, there will be no occurrence of a term greater than 1, so the scoring formula score becomes:

When the BM25 algorithm is used for search relevance score, the main operations are: perform morpheme analysis on Query to generate morpheme qi; then, for each search result D, calculate the correlation score between each morpheme qi and D, and finally, The correlation score of qi relative to D is weighted and summed to obtain the correlation score of Query and D. The general formula for transcribing the above formula into BM25 is as follows:

Among them, Q represents Query, and qi represents a morpheme parsed by Q (for Chinese, we can use the segmentation of Query as a morpheme analysis, and treat each word as a morpheme qi); d represents a search result document; Wi represents The weight of the morpheme; R(qi, d) represents the correlation score of the morpheme qi and the document d.

Wi is defined below to represent the weight of the correlation between a word and a document. There are many methods. If it is used as a general formula of BM25, it can be described as a form similar to IDF. The formula is as follows:

where N is the total number of documents in the index, and n(qi) is the number of documents that contain qi. According to the definition of IDF, it can be seen that for a given document set, the more documents that contain qi, the lower the weight of qi, that is to say, when many documents contain qi, the degree of discrimination of qi is not high, so the less important it is to use qi to judge the correlation. Let's look at the correlation score R(qi, d) between the morpheme qi and the document d. First, let's look at the general form of the correlation score in BM25:

Among them, k1.k2.b is the adjustment factor, usually set according to experience, generally k1=2, b=0.75; fi is the frequency of qi in d, and qfi is the frequency of qi in Query. dl is the length of document d, and avgdl is the average length of all documents. Since in most cases, qi only appears once in Query, that is, qfi=1, so the formula can be simplified as:

As can be seen from the definition of K, the function of parameter b is to adjust the size of the impact of document length on relevance. The larger the b, the greater the impact of the document length on the relevance score, and vice versa; the greater the relative length of the document, the greater the K value, and the smaller the relevance score. This can be understood as the greater the chance of containing qi when the document is longer. Therefore, under the same fi, the correlation between the long document and qi should be weaker than the correlation between the short document and qi.

In summary, the correlation score formula of the BM25 algorithm can be summarized as:

As can be seen from the formula of BM25, by using different morpheme analysis methods, morpheme weight determination methods, and morpheme-document correlation determination methods, we can derive different search relevance score calculation methods, which design the algorithm for us Provides greater flexibility.

(3) Sorting based on function

Many sorting models are based on functions. For example, the decay function can perform fractional decay and weight reduction based on dimensions such as time and number. Common decay functions such as gauss decay and linear decay.

gauss decay: a decay function derived from a Gaussian function, its formula is defined as follows:

The fraction of this formula will be attenuated after exceeding origin+/-offset; when the scale (attenuation limit) and decay (the degree of attenuation) are given, the shape of the attenuation function is also given, so the calculation method of σ is as follows:

σ ² =-scale ² /(2×ln(decay))

Exponential decay:

S(doc)=exp(λ×max(0, |fieldvalue _doc -origin|-offset))

Similarly, the score of this formula will be attenuated after exceeding origin+/-offset; when the scale (attenuation limit) and decay (attenuation degree) are given, the shape of the decay function is also given, so the calculation method of λ is as follows :

λ=ln(decay)/scale

Linear decay:

where s is calculated as follows:

s=scale/(1.0-decay));

Compared with the previous two formulas, when the fieldvalue exceeds twice the scale, the score of this formula will become 0.

2. Retrieval sorting algorithm.

The essence of the retrieval system is to achieve semantic-level matching based on the understanding of Query and patents, and then sort the matching candidate results. The sorting is divided into two stages: recall and fine sorting. The recall has ended, and the candidate results of TopN are returned. In the fine sorting stage, the results of TopN are accurately sorted based on the machine learning sorting model and the results of TopM (N>M) are selected. , referred to as LTR (learning to rank).

The following describes in detail the use of latent semantic analysis and association rule mining to construct the retrieval association model mentioned above.

1. Latent semantic analysis.

Latent Semantic Analysis (LSA) reduces synonymous noise by introducing a concept space. LSA exploits the contextual relevance of words, i.e. words that appear in similar contexts are considered to be similar in usage and meaning. To implement the LSA idea, first construct the word-document matrix:

A=|a _ij |m×n;

m represents the total number of vocabulary, n represents the number of documents, where a _ij is a non-negative value, indicating the weight of the i-th word appearing in the j-th document. Different words correspond to different rows of matrix A, and each document corresponds to a column of matrix A. Usually aij has to consider the contributions from two aspects, namely the local weight L(i,j) and the global weight C(i,j). In the VSM model, the local weights L(i,j) and the global weights C(i,j) have different weight calculation methods, such as IDF, TFIDF and so on. Since each word only appears in a small number of documents, A is usually a high-order sparse matrix. Let the i-th and j-th words correspond to the i-th row and the j-th row of the word-document matrix, respectively, denoted as ti=(ai1, ai2,..., ain) and tj=(aj1,aj2,...,ajn) , whose similarity is defined as

Before calculating the similarity, aij is usually converted into log(aij+1), and then divided by its entropy, so that the preprocessing can take the context of the word into account and highlight the context of the word in the article. After information entropy transformation, the ordered word-document matrix A'=a'ij m×n is obtained, where:

The theoretical basis of latent semantic analysis is the singular value decomposition (SVD) of the matrix, which is a commonly used method in mathematical statistics. After the word-document matrix A' is established, the k-rank approximation matrix A'k(k<<min(m,n)) of A' is calculated by singular value decomposition. After singular value decomposition, the matrix A' can be expressed as the sum of three matrices. product:

A'=U∑V ^T ;

In the formula, U and V are the left and right singular vector matrices corresponding to the singular values of A' respectively; Σ is the standard form; VT is the rank of V; the singular values of A' are arranged in decreasing order to form a diagonal matrix Σk, take U and The first k columns of V construct a k-rank approximation matrix of A':

In the formula, the column vectors of Uk and Vk are both orthogonal vectors. Assuming that the rank of A' is r, there are:

U ^T U=V ^T V=I _r (I _r is a unit matrix of order r×r);

A' _k is used to approximate the original word-document matrix A', and the row vectors in U _k and V _k are used as word vectors and document vectors respectively. On this basis, text classification and various other document processing are performed, which is the implicit semantics. indexing technology. Although LSI also uses the words contained in the document to represent the semantics of the document, the LSI model does not regard all the words in the document as a reliable representation of the document concept. Since the diversity of words in a document largely obscures the semantic structure of the document, LSI reduces the "noise" factor contained in the original word-document matrix by singular value decomposition and k-rank approximation matrix, which makes it more prominent. The semantic relationship between words and documents; on the other hand, the vector space of words and documents is greatly reduced, which can improve the efficiency of text mining.

2. Association rule mining.

并且X∩Y＝Φ规则X→Y在事务集D 中的支持度是事务集中包含X和Y的事务数与所有事务数之比，记为support(X →Y)，即：Association rules are one of the main mining techniques in data mining, which can find potentially useful associations or correlations from massive data. Let I={i1,i2,...,im} be the set of items, and denote D as the set of transactions T, where transaction T is the set of items, and T∈I. There is a unique identifier corresponding to each transaction, denoted as set TID. Let X be a set of items in I, if X ∈ T, then transaction T is said to contain X. An association rule is an implication of the form X→Y, where

And X∩Y=ΦThe support of rule X→Y in transaction set D is the ratio of the number of transactions containing X and Y to the number of all transactions in the transaction set, denoted as support(X→Y), that is:

The credibility of the rule X→Y in the transaction set refers to the ratio of the number of transactions containing X and Y to the number of transactions containing X, denoted as confidence(X→Y), that is:

Given a transaction set D, the problem of association rule mining is to generate association rules whose support and credibility are greater than the minimum support and minimum credibility given by the user, respectively. Common algorithms include Apriori algorithm and FP-Growth algorithm.

For associated word mining, suppose the mined association rules are in the form of {ti→tj,s,c}, indicating that the word ti appears in the document, then the support degree of the word tj appearing in the same document is s (0≤s≤1) , the confidence level is c (0≤c≤1). If the support and confidence are greater than the specified threshold, it can be considered that their correlation is so large that even ignoring words will not cause information loss, which ensures the independence of words.

3. Algorithmic process.

According to the above principle analysis, the construction algorithm of synonyms and associated word sets is described as follows:

Algorithm 1: The synonym set construction algorithm.

enter:

1) Training document set;

2) Concept space dimension k;

3) The number N of feature words is reserved for synonymous merger.

output:

1) A feature word set with a total of N elements;

2) Merge scheme set (decide that the merged synonyms should be merged into which reserved feature words).

step:

1) Construct a word-document matrix according to the condition that the training document contains words;

2) Perform sequential processing on the word-document matrix;

3) Use SVD to perform singular value decomposition on the word-document matrix to obtain left and right singular vectors and singular value canonical forms of the word-document matrix;

4) Keep the first k columns of left singular vector data, and clear all other column data to zero;

5) Retain the first k data on the diagonal of the singular value standard type, and clear all other diagonal data to zero;

6) Retain the data of the first k columns of the right singular vector, and clear all other column data to zero;

7) Multiply the left and right singular vectors after zeroing with the singular value standard type to obtain a new word-document matrix;

8) Perform sequential processing on the new word-document matrix;

9) When the number of words exceeds N, perform steps 10 to 14 to merge synonyms;

10) Find two feature words with the largest similarity in the feature word set;

11) Delete any one of the feature word pairs with the greatest similarity in the feature word set;

12) Find the merging scheme containing any feature word in the feature word pair with the greatest similarity in the merging scheme set;

13) If found, add another feature word to the merged feature word set of the merge scheme;

14) If the merging scheme cannot be matched, construct a merging scheme with these two feature words and put it into the set of merging schemes.

Algorithm 2: Associated word set construction algorithm.

enter:

1) Training document set;

2) Association merge threshold: support s, confidence c.

output:

1) Feature word set after association and merge;

2) A set of merging schemes after association and merging.

step:

1) Use the Aprior algorithm to obtain all single-dimensional associations with support and confidence greater than s and c, respectively;

2) Perform steps 3 to 6 for each single-dimensional association rule with support and confidence greater than s and c, respectively, and perform association merging;

3) Delete the feature word in the right part of the association rule in the feature word set;

4) Find the merge scheme containing the feature words on either side of the association rule in the merge scheme set;

5) If found, add another feature word to the merged feature word set of the merge plan;

6) If the merging scheme cannot be matched, construct a merging scheme with the two characteristic words in the left and right parts of the association rule, and put it into the merging scheme set.

Practice has proved that the construction of retrieval association model can be effectively realized through latent semantic analysis and association rule mining.

To sum up, the method for determining the similarity of documents based on semantic analysis provided by the embodiments of the present invention can accurately determine the weights of different parts of documents based on semantic analysis, and then accurately determine the similarity of documents.

The embodiments of the present invention also provide a device for determining the similarity of documents based on semantic analysis, as described in the following embodiments. Since the principle of the device for solving the problem is similar to the method for determining the similarity of documents based on semantic analysis, the implementation of the device can refer to the implementation of the method for determining the similarity of documents based on semantic analysis, and the repetition will not be repeated.

FIG. 5 is a schematic structural diagram of a device for determining the similarity of documents based on semantic analysis in an embodiment of the present invention. As shown in FIG. 5 , the device includes:

A dividing unit 01 is used to divide each document to be compared into a plurality of parts;

A semantic analysis unit 02, configured to perform semantic analysis on each part to obtain a semantic analysis result of each part;

A weight value determination unit 03, configured to determine the weight value of each part of each document to be compared according to the semantic analysis result of each part;

The processing unit 04 is configured to obtain the weighted average result of each document to be compared according to the weight value of each part of each document to be compared;

The similarity determination unit 05 is configured to determine the similarity between the documents to be compared according to the weighted average result of each document to be compared.

In one embodiment, as shown in FIG. 6 , the semantic analysis unit 02 may include:

The word segmentation processing module 021 is used to perform word segmentation processing on each part to obtain a plurality of keywords corresponding to each part;

The feature extraction module 022 is used to extract multiple types of key features from each part according to a plurality of keywords corresponding to each part and a preset document feature extraction strategy to form a feature set corresponding to each part;

The feature evaluation module 024 is configured to perform word-level, syntactic-level and text-level semantic analysis on each part according to the feature set corresponding to each part, and obtain the semantic analysis result of each part.

In one embodiment, as shown in FIG. 7 , the above-mentioned semantic analysis unit 02 may further include a feature dimension reduction module 023 for using principal component analysis, linear discriminant analysis and mutual information The feature set performs feature screening and combination to obtain the feature set corresponding to each part after feature dimension reduction processing;

The feature evaluation module 024 is specifically configured to: perform word-level, syntactic-level and chapter-level semantic analysis on each part according to the feature set corresponding to each part after feature dimension reduction processing, and obtain the semantic analysis result of each part.

In one embodiment, the plurality of types of key characteristics may include: document static characteristics, characteristics of documents associated with the query, and characteristics of the query.

In one embodiment, the preset document feature extraction strategy may include: according to the frequency of the keyword appearing in the document, the inverse document frequency of the keyword, the part of speech of the keyword, whether the keyword is a professional word, the occurrence of the keyword The position in the literature, the text-rank value of the keyword, the information entropy value of the keyword, the word vector of the keyword and the overall deviation value, the length of the keyword, the keyword as a component of the sentence, whether the keyword is further divided into sub-sections One or any combination of keywords, the length of the first occurrence and the last occurrence of the keywords in the document, and the deviation of the keyword distribution, are used for document feature extraction.

In one embodiment, when the keywords can be further divided into sub-keywords, the preset document feature extraction strategy may further include: according to the word frequency of the sub-keywords-inverse document frequency, the part-of-speech of the sub-keywords, Whether the sub-keyword is one of the specialized words or any combination is used for feature extraction of documents.

The acquisition, storage, use, and processing of data in the technical solution of this application are in compliance with the relevant provisions of national laws and regulations.

An embodiment of the present invention further provides a computer device, including a memory, a processor, and a computer program stored in the memory and running on the processor, when the processor executes the computer program, the above-mentioned documents similar to those based on semantic analysis are implemented degree determination method.

An embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the above-mentioned method for determining the similarity of documents based on semantic analysis is implemented.

An embodiment of the present invention further provides a computer program product, where the computer program product includes a computer program, and when the computer program is executed by a processor, implements the above-mentioned method for determining the similarity of documents based on semantic analysis.

In the embodiment of the present invention, the solution for determining the similarity of documents based on semantic analysis is different from the prior art, which presets different fixed weights for different parts of the documents based on experience, and then determines the similarity of documents. Compared with the technical solutions for the problem of inaccurate similarity determination, by: dividing each document to be compared into multiple parts; performing semantic analysis on each part to obtain the semantic analysis result of each part; according to the semantic analysis result of each part, Determine the weight value of each part of each document to be compared; obtain the weighted average result of each document to be compared according to the weight value of each part of each document to be compared; By comparing the similarity between documents, the weight of different parts of documents can be accurately determined based on semantic analysis, and then the similarity of documents can be accurately determined.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each process and/or block in the flowchart illustrations and/or block diagrams, and combinations of processes and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flows of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above-mentioned specific embodiments are only specific embodiments of the present invention, and are not intended to limit the scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims (10)

1. a document similarity determination method based on semantic analysis, is characterized in that, comprises: Divide each document to be compared into multiple sections; Perform semantic analysis on each part to obtain the semantic analysis result of each part; According to the semantic analysis result of each part, determine the weight value of each part of each document to be compared; According to the weight value of each part of each document to be compared, the weighted average result of each document to be compared is obtained; According to the weighted average result of each document to be compared, the similarity between documents to be compared is determined. 2. The method for determining the similarity of documents based on semantic analysis as claimed in claim 1, wherein the semantic analysis of each part is performed to obtain the semantic analysis result of each part, comprising: Perform word segmentation on each part to obtain multiple keywords corresponding to each part; According to the multiple keywords corresponding to each part and the preset literature feature extraction strategy, multiple types of key features are extracted from each part to form a feature set corresponding to each part; According to the feature set corresponding to each part, perform word-level, syntactic-level and chapter-level semantic analysis on each part, and obtain the semantic analysis result of each part. 3. the document similarity determination method based on semantic analysis as claimed in claim 2, is characterized in that, also comprises: utilize principal component analysis method, linear discriminant analysis method and mutual information method, carry out feature set to each part corresponding feature set to obtain the feature set corresponding to each part after feature dimensionality reduction processing; According to the feature set corresponding to each part, perform word-level, syntactic-level and chapter-level semantic analysis on each part, and obtain the semantic analysis result of each part, including: according to the feature set corresponding to each part after feature dimension reduction processing, for each part One part performs word-level, syntactic-level and text-level semantic analysis, and obtains the semantic analysis results of each part. 4 . The method for determining the similarity of documents based on semantic analysis according to claim 2 , wherein the multiple types of key features include: static features of documents, features associated with documents and queries, and features of queries. 5 . 5. The method for determining the similarity of documents based on semantic analysis according to claim 2, wherein the preset document feature extraction strategy comprises: according to the frequency of keywords appearing in documents, the inverse document frequency of keywords , the part of speech of the keyword, whether the keyword is a professional word, the position of the keyword in the literature, the text-rank value of the keyword, the information entropy value of the keyword, the word vector of the keyword and the overall deviation value, the length of the keyword , the keyword is used as a component of the sentence, whether the keyword is further divided into sub-keywords, the length of the first occurrence and the last occurrence of the keyword in the literature, one or any combination of the keyword distribution deviation, and the literature Feature extraction. 6. The method for determining the similarity of documents based on semantic analysis as claimed in claim 5, wherein when the keywords can be divided into sub-keywords, the preset document feature extraction strategy further comprises: according to The word frequency of the sub-keyword - the inverse document frequency, the part-of-speech of the sub-keyword, whether the sub-keyword is one of the professional words or any combination is used for document feature extraction. 7. A device for determining the similarity of documents based on semantic analysis, comprising: A division unit, which is used to divide each document to be compared into a plurality of parts; The semantic analysis unit is used to perform semantic analysis on each part to obtain the semantic analysis result of each part; a weight value determination unit, configured to determine the weight value of each part of each document to be compared according to the semantic analysis result of each part; a processing unit, configured to obtain a weighted average result of each document to be compared according to the weight value of each part of each document to be compared; The similarity determination unit is configured to determine the similarity between the documents to be compared according to the weighted average result of each document to be compared. 8. A computer device comprising a memory, a processor and a computer program stored on the memory and running on the processor, wherein the processor implements any one of claims 1 to 6 when the processor executes the computer program the method. 9 . A computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method of any one of claims 1 to 6 is implemented. 10 . 10. A computer program product, characterized in that the computer program product comprises a computer program, which implements the method of any one of claims 1 to 6 when the computer program is executed by a processor.

Images