RU2681356C1 - Classifier training used for extracting information from texts in natural language - Google Patents

Abstract

FIELD: examination of documents.SUBSTANCE: invention relates to the extraction of facts from texts in natural languages. First set of information objects is extracted from the natural language text. Second set of information objects is extracted from the natural language text. Intermediate list of information objects is generated, including at least a subset of the first set of information objects and at least a subset of the second set of information objects. Set of conflicting information objects in the intermediate list of information objects is identified, where the first information object from the set of conflicting information objects belongs to the first set of information objects, and the second information object from the set of conflicting information objects belongs to the second set of information objects. Final list of information objects extracted from natural language text is generated, by means of applying the function of arbitration of the conflicting objects to the set of conflicting information objects, which performs at least one of the following actions: changing the first information object, deleting the first information object or merging two or more information objects from the set of conflicting information objects.EFFECT: technical result consists in increasing the efficiency and quality of information extraction.20 cl, 16 dwg

Description

FIELD OF TECHNOLOGY

[0001] The present invention generally relates to computing systems, and more specifically, to systems and methods for processing a natural language.

BACKGROUND

[0002] Information retrieval may include analysis of natural language text to identify and classify information objects according to a given set of categories (for example, names of individuals, organizations, places, expressions of time, quantity, monetary amounts, percent, etc.). Information retrieval may further include determining the relationship between recognized named entities and / or information objects.

SUMMARY OF THE INVENTION

[0003] In accordance with one or more embodiments of the present invention, an example of a method for extracting information from a natural language text may include: extracting by a computer system a first plurality of information objects from a natural language text; extracting from the text in the natural language of the second set of information objects; creating an intermediate list of information objects, including at least a subset of the first set of information objects and at least a subset of the second set of information objects; identifying in the intermediate list of information objects a plurality of conflicting information objects, in which a first information object of a plurality of conflicting information objects belongs to a first plurality of information objects, and a second information object of a plurality of conflicting information objects belongs to a second plurality of information objects; and creating a final list of information objects extracted from the text in a natural language by applying to the set of conflicting information objects the function of “arbitration” of conflicting objects, which performs at least one of the following: changing the first information object, deleting the first information object, or merging two and more information objects from many conflicting information objects.

[0004] In accordance with one or more embodiments of the present invention, an example system for extracting information from natural language text may include: memory; a processor associated with the specified memory, and this processor is configured to: extract from the text in natural language the first set of information objects; extracting from the text in the natural language of the second set of information objects; creating an intermediate list of information objects, including at least a subset of the first set of information objects and at least a subset of the second set of information objects; identifying in the intermediate list of information objects a plurality of conflicting information objects, in which a first information object of a plurality of conflicting information objects belongs to a first set of information objects, and a second information object of a plurality of conflicting information objects belongs to a second set of information objects; and creating a final list of information objects extracted from natural language text by applying to the set of conflicting information objects the function of “arbitration” of conflicting information objects, which performs at least one of the following: changing the first information object or merging two or more information objects from many conflicting information objects ..

[0005] In accordance with one or more embodiments of the present invention, an example of a permanent computer-readable storage medium may include executable instructions that, when executed by a computer system, result in the computer system performing operations, including: extracting from a natural language text a first set of information objects ; extracting from the text in the natural language of the second set of information objects; creating an intermediate list of information objects, including at least a subset of the first set of information objects and at least a subset of the second set of information objects; identifying in the intermediate list of information objects a plurality of conflicting information objects, in which a first information object of a plurality of conflicting information objects belongs to a first set of information objects, and a second information object of a plurality of conflicting information objects belongs to a second set of information objects; and creating a final list of information objects extracted from natural language text by applying to the set of conflicting information objects the function of “arbitration” of conflicting information objects, which performs at least one of the following: changing the first information object or merging two or more information objects from multiple conflicting information objects

BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The present invention is illustrated by way of examples, and not by way of limitation, and can be better understood by considering the following description of preferred embodiments in conjunction with the drawings, in which:

[0007] In FIG. 1 is a flowchart of one illustrative example of a method for extracting information using several extraction methods in accordance with one or more embodiments of the present invention;

[0008] In FIG. 2 schematically shows an example of “arbitration” of conflicting entities that may be performed in accordance with one or more embodiments of the present invention, in accordance with one or more embodiments of the present invention;

[0009] In FIG. 3 is a flowchart of one example of a classifier training method used to extract information from natural language texts in accordance with one or more embodiments of the present invention;

[00010] In FIG. 4 is a flowchart of another example of a classifier training method used to extract information from natural language texts in accordance with one or more embodiments of the present invention;

[00011] In FIG. 5 is a flowchart of one illustrative example of a method 400 for semantic-syntactic analysis of a sentence in a natural language in accordance with one or more embodiments of the present invention;

[00012] In FIG. 6 schematically shows an example of the lexical-morphological structure of a sentence in accordance with one or more embodiments of the present invention;

[00013] In FIG. 6 schematically shows language descriptions representing a natural language model in accordance with one or more embodiments of the present invention;

[00014] In FIG. 8 schematically shows examples of morphological descriptions in accordance with one or more embodiments of the present invention;

[00015] In FIG. 9 schematically shows examples of syntactic descriptions in accordance with one or more embodiments of the present invention;

[00016] In FIG. 10 schematically shows examples of semantic descriptions in accordance with one or more embodiments of the present invention;

[00017] In FIG. 11 schematically shows examples of lexical descriptions in accordance with one or more embodiments of the present invention;

[00018] In FIG. 12 schematically shows examples of data structures that can be used within one or more methods implemented in accordance with one or more embodiments of the present invention;

[00019] In FIG. 13 schematically shows an example of a graph of generalized components in accordance with one or more embodiments of the present invention;

[00020] FIG. 14 is an example of a syntax structure corresponding to the sentence of FIG. 13;

[00021] In FIG. 15 shows a semantic structure corresponding to the syntactic structure shown in FIG. fourteen;

[00022] FIG. 16 is a diagram of an example computer system implementing the methods of the present invention.

DESCRIPTION OF PREFERRED EMBODIMENTS

[00023] This document describes methods and systems for extracting information from natural language texts. The systems and methods presented in this document can find application in a variety of situations where natural language text processing is required, in particular, it can be machine translation, semantic indexing, semantic search (including multilingual semantic search), classification of documents, search and submission of electronic documents (e-discovery), etc.

[00024] As used herein, the term "computing system" means a data processing device equipped with a universal or specialized processor, memory, and at least one communication interface. Examples of computing systems that can use the methods described in this document are, in particular, desktop computers, laptops, tablet computers and smartphones.

[00025] “Ontology” in this document means a model that represents objects belonging to a certain field of knowledge (domain), and the relationship between these objects. An ontology may include definitions of a certain set of classes, where each class corresponds to the concept of a subject area. Each class definition may include definitions of one or more objects assigned to that class. According to generally accepted terminology, an ontology class can also be called a “concept”, and an object belonging to the class can mean an instance of this concept. In some implementations of the invention, the class may be the ancestor or descendant of another class. In certain embodiments, an object may be associated with two or more classes.

[00026] The definition of each class may further include one or more definitions of the relations of one or more objects assigned to this class. Relationships define various types of interactions between related objects. In some implementations of the invention, various relationships can be organized in a general taxonomy, for example, the relationship of “paternity” and “motherhood” can be included in the more general relationship of “being a parent,” which in turn can be included in the more general relationship of “being an ancestor” .

[00027] The attribute of an information object may be represented by another information object (for example, the place of birth of a person may be represented by an information object of class "City"; and the employer of a person may be represented by an object of class "Company"). A class definition may include one or more restrictions associated with certain attributes of an object of this class, for example, an attribute of cardinality restriction (for example, a person can have only one place of birth) or type restrictions (for example, a given attribute can be represented by objects of a specified set of classes )

[00028] Information retrieval may include natural language text analysis to identify and classify information objects referenced by natural language text fragments. Text fragments that refer to an information object are called "object annotations." Text fragments that refer to an attribute of an information object are called "attribute annotations." An annotation can be determined by its position in a text in a natural language, including the position of the beginning (word) and the position of the end (word).

[00029] When retrieving information, a classification of identified information objects may occur in accordance with a predefined set of categories (for example, names of people, names of organizations, places, expressions of time, quantity, money, percent, etc.). Such categories can be represented by concepts of a predefined or dynamically formed ontology. Information retrieval may further include determining the relationship between recognized named entities and / or information objects. Examples of such relationships can be the work of person X in organizational unit Y, the location of object X in geoposition Y, the acquisition by organizational unit X of organizational unit Y, etc. Such relationships can be expressed in fragments in a natural language, which can contain many words from one or more sentences. Relationships of information objects can be expressed by attributes of an information object that refer to other information objects (for example, an information object “person” can have such attributes as “place of birth”, “place of residence” or “employer”, each of which is represented by a link to information object of the corresponding class).

[00030] The extracted information that can be represented by an RDF graph can be used to perform various operations and tasks of natural language processing, including machine translation, semantic indexing, semantic search, document classification, search and presentation of electronic documents (e-discovery) and etc.

[00031] Information extraction can be performed using various extraction methods using variable sets of rules, automatic classification methods (also called “machine learning classifiers”), heuristic approaches and / or a combination of these methods. Information extraction methods can be based on ontologies and (or) other document models for the analysis of morphological, lexical, syntactic, semantic and (or) other attributes of texts in natural language.

[00032] In an illustrative example, extracting information may include applying a variety of production rules, for example, matching with a pattern defined on the left side of a production rule, with a semantic structure representing a piece of text in natural language, can trigger an action on the right side of a production rule that can assign specific values to one or more attributes of the information object. In another illustrative example, a classifier based on machine learning can determine the degree to which a given fragment of a text in a natural language is connected (for example, represented by a set of morphological, lexical, syntactic, semantic and (or) other attributes) with a certain class of information objects. Classifier parameters can be determined or changed as a result of further training of the classifier using pre-existing or dynamically created training data samples based on which the classes of information objects are correlated with morphological, lexical, syntactic, semantic and (or) other text attributes in natural language. Classification methods may include differential evolution methods, genetic algorithms, random forest methods, neural networks, etc.

[00033] Different methods of extracting information may have different performance characteristics, such as computational accuracy, completeness, and / or computational complexity. Accordingly, different methods of extracting information can produce different results when applied to the same text in a natural language. Certain information objects extracted using two or more methods of extracting information may conflict with each other, for example, the same word in a natural language can be recognized as a reference to a geographical location or a person’s name.

[00034] The present invention improves the efficiency and quality of information extraction by creating systems and methods that use various combinations of information extraction methods with subsequent determination and resolution of conflicts among the extracted information objects, thus achieving a result that exceeds the quality and efficiency of traditionally used methods. In an illustrative example, a computing system that implements this method can use two or more methods of extracting information for the same natural language text. Then, the computing system can detect the sets (for example, pairs, triplets, quadruplets, etc.) of potentially conflicting objects extracted by various methods, for example, by detecting overlapping text annotations and (or) violations of various restrictions associated with attributes of information objects (for example , attribute of cardinality restriction). After determining the set of conflicting objects, the computing system can attempt to resolve the conflict by applying the function of “arbitration” of conflicting objects, which can analyze morphological, lexical, syntactic, semantic and (or) other attributes of conflicting information objects in order to obtain a result confirming one or more potentially conflicting objects, changing the attributes of one or more potentially conflicting objects, deleting one or more potentially conflicting objects and / or merging one or more potentially conflicting objects.

[00035] In certain embodiments, the arbitration function of conflicting information objects may apply one or more customizable rules for evaluating logical conditions defined for morphological, lexical, syntactic, semantic and (or) other attributes of the extracted information objects. Alternatively, the “arbitration” function of conflicting objects may include one or more classifiers based on machine learning. In an illustrative example, the classifier used by the “arbitration” function of conflicting objects can provide the degree of validity of one or more possible results of the “arbitration” (for example, merging possible conflicting objects into a single object, confirming one or more extracted objects, changing one or more extracted objects and (or) deletion of one or more extracted objects). In another illustrative example, the classifier used by the “arbitration” function of conflicting objects can ensure the similarity of two or more information objects representing the same real-life object. In yet another illustrative example, a classifier used by the arbitrage function of conflicting objects can create degrees of confidence associated with each information object from a plurality of conflicting information objects.

[00036] Classifiers used by the “arbitration” function of conflicting objects can be trained on the basis of pre-existing or dynamically created training data samples, based on which the results of the “arbitration” can be correlated with morphological, lexical, syntactic, semantic and (or) other attributes of extracted information objects. Classification methods may include differential evolution methods, genetic algorithms, random forest methods, neural networks, etc.

[00037] In certain embodiments, training data sets for classifiers based on machine learning can be created by applying rules-based methods for selecting texts in natural language and / or by user verification of classification results obtained through the use of trained classifiers. In an illustrative example, classifiers can iteratively retrain on the basis of classification results verified by the user, due to which the quality of classification is gradually improved with each iteration of training. The contours of such training based on feedback can be applied to classifiers that perform the initial extraction of information objects, as well as classifiers that implement the functions of “arbitration” of conflicting objects, thereby gradually increasing the overall quality of information extraction.

[00038] In some embodiments, the definition of the classifier used to extract the information may include one or more parameters, the values of which can be adjusted using machine learning methods, and may also include one or more hyperparameters, the values of which can be determined by some external to machine learning methods by operations or processes. In an illustrative example, the values of one or more hyperparameters of the information extraction classifier can be iteratively controlled by iteratively performing the operations of training the information extraction classifier, training the conflict arbitration classifier, performing the conflict arbitration function, and changing the hyperparameter values, thereby iteratively optimizing the quality indicator using which evaluates the final list of extracted information objects, as described in more detail on Also in this document.

[00039] Systems and methods operating in accordance with one or more embodiments of the present invention can be used to perform various natural language processing operations, such as machine translation, semantic search, classification and clustering of objects, etc. In some implementations, the extracted information can be visually represented using a graphical user interface, for example, by displaying the identifiers of classes of information objects in visual comparison with the corresponding fragments of the text of a natural language.

[00040] Various aspects of the above methods and systems are described in detail later in this document by way of examples, and not by way of limitation.

[00041] In FIG. 1 is a flowchart of one illustrative example of a method for extracting information using several extraction methods in accordance with one or more embodiments of the present invention. The method 100 and / or each of its individual functions, standard programs, subprograms, or operations can be performed by one or more processors of a computing system (eg, computing system 1000 in FIG. 16) that implements this method. In some embodiments, method 100 may be performed in a single processing stream. In an alternative approach, method 100 may be implemented using two or more processing streams, with each stream implementing one or more separate functions, procedures, subroutines, or actions of this method. In one illustrative example, processing streams that implement method 100 can be synchronized (for example, using semaphores, critical sections, and / or other thread synchronization mechanisms). In an alternative approach, the processing threads implementing the method 100 may be performed asynchronously with respect to each other.

[00042] In step 110, a computing system implementing the method may receive natural language text. In an illustrative example, a computing device can receive natural language text in the form of an electronic document, which can be obtained by scanning or by using another image method from a paper document, followed by optical character recognition (OCR) to obtain the text of the document. In another illustrative example, a computing system may receive natural language text in the form of one or more formatted files, for example, text editor files, email messages, digital data files, etc.

[00043] In step 120, the computing system can apply the first information extraction method to the natural language text. In certain implementations, the first method of extracting information can use variable sets of rules, automatic classification methods (also called “machine learning classifiers”), heuristic approaches and / or a combination of these methods. When applying the first method of extracting information to a text in a natural language, many information objects of one or more types of information objects can be created.

[00044] In one illustrative example, the first method of extracting information can use a variable set of production rules to interpret the set of semantic-syntactic structures obtained as a result of semantic-syntactic analysis of text in natural language. Each semantic structure can represent a sentence in the corresponding natural language. Each semantic structure can be represented by an acyclic graph that includes many nodes corresponding to semantic classes and many arcs corresponding to semantic relations, which is described in more detail below with reference to FIG. fifteen.

[00045] Production rules applied to a variety of semantic structures may include interpretation rules and identification rules. An interpretation rule may contain the left part represented by a set of logical expressions defined on one or more semantic structure templates and the right part represented by one or more statements regarding information objects representing entities referred to in a natural language text. The semantic structure template may include certain elements of the semantic structure (for example, a relationship with a specific lexical / semantic class, a relationship with a specific surface or depth position at a specified location in the semantic structure, the presence of ancestor-child relationships between certain nodes of the semantic structure, and a number of levels of the semantic hierarchy between parent and child classes in a specific ontology, the presence of a common ancestor for certain nodes of the semantic structure, the presence of PARTICULAR grammemes or semantics, etc.). Relations between elements of semantic structures can be specified using one or more logical expressions (conjunction, disjunction, and negation) and (or) operations characterizing the mutual arrangement of nodes in a syntactic-semantic tree. As an illustrative example, such an operation can check whether a node belongs to a subtree of another node.

[00046] The right part of the production rule may assign and (or) change the values of one or more attributes of information objects that display morphological, lexical, syntactic, semantic and (or) other attributes of a fragment of text in a natural language. In one illustrative example, the right side of the interpretation rule may contain a statement linking a piece of natural language text to a class of information objects.

[00047] An identification rule can be used to establish an associative relationship for a pair of information objects that represent the same entity in the real world. An identification rule is a production rule, the left part of which contains one or more logical expressions pointing to the nodes of the semantic tree corresponding to information objects. If the specified pair of information objects satisfies the conditions specified by logical expressions, then the information objects merge into one information object.

[00048] In an illustrative example, an information extraction method may use one or more machine learning classifiers. Each machine learning classifier can determine the degree of connectedness of a given fragment of text in a natural language (for example, represented by a set of morphological, lexical, syntactic, semantic and (or) other attributes) with a specific class of information objects. A computer system can iteratively process many fragments of text in a natural language and determine for each text fragment the degree of its connection with one or more classes of information objects. Classification methods may include differential evolution methods, genetic algorithms, random forest methods, neural networks, etc.

[00049] Classifiers can be trained using pre-existing and (or) dynamically created training data samples based on which the correlation of classes of information objects with lexical, syntactic, semantic and (or) other attributes of a text in a natural language is performed. A training data set may include one or more natural language texts accompanied by metadata that define some information objects, their classification, attributes, and corresponding text annotations. In an illustrative example, metadata may be represented by markup associated with natural language text. In certain implementations, the training data set can be iteratively improved by adding new natural language texts, accompanied by user-confirmed metadata. Validation of metadata may include receiving user input through a graphical user interface that confirms or corrects the extracted information objects and their attributes.

[00050] The extracted information objects may be represented as an RDF graph. RDF (Resource Definition Framework) assigns a unique identifier to each information object and stores information about such an object in the form of sets of three SPO elements (triplets), where S means “subject” and contains the identifier of the object, P means “predicate” and defines some property of this object, and O means "object" and stores the value of the considered property of this object. This value can be either a primitive data type (examples: string, number, Boolean (logical) value), or the identifier of another object. In one illustrative example, an SPO triplet can associate a fragment from a text in natural language with a category of information objects.

[00051] In certain embodiments, information retrieval methods can be based on ontologies and (or) other document models for analyzing lexical, syntactic, semantic and (or) other attributes of a text in natural language. Thus, the information retrieval operations indicated in step 120 may be preceded by one or more document pre-processing operations performed to determine the structure of the document and determine the corresponding model of the document. In one illustrative example, a document structure may include a hierarchical multi-level structure in which sections of a document are separated by headers and subheadings. In another illustrative example, a document structure may include one or more tables containing several rows and columns, at least some of which may be associated with headings, which in turn may be organized into a multi-level hierarchy. In another illustrative example, a document structure may include a page structure comprising a page header, a page body, and / or a page footer. In another illustrative example, a document structure may include certain text fields associated with predetermined types of information, such as a signature field, a date field, an address field, a name field, etc. The computing system 100 that implements this method can interpret the structure of the document to obtain certain information about the structure of the document, which can be used to expand the text information contained in this document. In some implementations of the invention, when analyzing structured documents, a computing system may use various auxiliary ontologies containing classes and concepts that reflect the specific structure of the document. The auxiliary ontology classes can be associated with certain production rules and / or classifier functions, which can be applied to several semantic structures obtained by semantic-syntactic analysis of the corresponding document for inclusion in the resulting set of semantic structures of certain information transmitted by the structure of this document.

[00052] In step 130, the computing system can apply a second method of extracting information to a natural language text that is different from the first method of extracting information. In certain implementations, the second method of extracting information can use variable sets of rules, automatic classification methods (also called “machine learning classifiers”), heuristic approaches and / or a combination of these methods, which is described in more detail below with reference to step 120. When applying a second set of information objects can be created that can be different from the first set of information objects created by applying the first method of extracting information.

[00053] In FIG. 1 shows only two sets of information objects retrieved using two different methods of extracting information, however, various embodiments of method 100 may include two or more different methods of extracting information to extract two or more sets of information objects.

[00054] At step 140, the computing system may create an intermediate 250 list of information objects, which may include at least a subset of each set of information objects extracted by applying various methods of extracting information in steps 120-130. In an illustrative example, a computing system may include in an interim list information objects associated with one or more predefined classes of information objects, with the exception of information objects associated with other classes. In another illustrative example, a computing system may exclude from the interim list information objects of one or more predetermined classes of information objects. As described above, an interim list may include one or more sets of potentially conflicting information objects, the joint existence of which violates one or more specified rules (for example, overlapping text annotations or cardinality restrictions). An intermediate list can be represented by an RDF graph created by merging RDF graphs representing the corresponding sets of extracted information objects.

[00055] At step 150, the computing system can determine one or more sets of conflicting information objects, each set including two or more information objects, at least two of which were extracted by various methods of extracting information. In an illustrative example, a plurality of conflicting information objects may include two information objects, wherein the first information object relates to the first set of information objects extracted by the first information extraction method, and the second information object relates to the second set of information objects extracted by the second information extraction method.

[00056] In an illustrative example, identifying potentially conflicting information objects may include identifying extracted information objects having at least partially overlapping text annotations. A computer system that implements this method can iteratively process an intermediate list of extracted information objects in order to identify sets of information objects (for example, pairs, triplets, quadruplets, etc.) with intersecting text annotations. In certain embodiments, a computing system may determine that two or more extracted information objects conflict if their respective annotations intersect at least in a threshold number of words.

[00057] In another illustrative example, the determination of potentially conflicting information objects may include the determination of various constraints associated with attributes of information objects. In certain embodiments, a computing system that implements this method can evaluate the logical conditions defined for attributes of information objects (for example, comparing an attribute of one information object with an attribute of another information object). In an illustrative example, a computing system can detect a violation of the cardinality limit by defining an information object that has a number of attributes of a certain type (for example, a person can have only one date of birth and one place of birth). Cardinality and other restrictions can be specified in the model of the document or ontology associated with the processed document.

[00058] At step 160, the computing system can apply the function of “arbitration” of conflicting objects for each identified set of conflicting information objects to resolve conflicts in the intermediate list of information objects, thereby creating a final list of information objects extracted from natural language texts. For each set of conflicting information objects, the “arbitration” function of conflicting objects can resolve the conflict by analyzing the morphological, lexical, syntactic, semantic and (or) other attributes of conflicting information objects in order to create a result confirming one or more potentially conflicting objects, changing one or more potentially conflicting objects, deleting one or more potentially conflicting objects and (or) merging one or more potentially conflicting objects. Based on the results of the function of "arbitration" of conflicting objects, the computing system can modify the RDF graph representing an intermediate list of extracted information objects, on the basis of which the final RDF graph is created.

[00059] FIG. 2 schematically shows an example of “arbitration” of conflicting entities that can be performed in accordance with one or more embodiments of the present invention. As shown in FIG. 2, when the first set of information extraction method 220A is applied to the natural language text 210, the first set of information objects 230A is created, and when the second method of information extraction 220V is applied to the text in natural language 210, the second set of information objects 230B is created. The first set of information objects 230A may include an information object 240A, which may be potentially conflicting with the information object 240B included in the second set of information objects 230B. When applying the function of “arbitration” of conflicting objects 250 to potentially conflicting information objects 260, one of the results of the “arbitration” of conflicting objects 260 can be created. The result of the “arbitration” of conflicting objects 260A includes confirmation of information objects 240A and 240B and changing at least some of their attributes (for example, by changing the class of the information object and corresponding resolution of the obvious conflict). The result of the “arbitration” of conflicting entities 260B includes the removal of the information entity 240B. At least some of the attributes of the remote information object 240B may be copied to the remaining information object 240A. The result of the “arbitration” of conflicting objects 260C includes the merger of information objects 240A and 240B into a new information object 240C.

[00060] In certain embodiments, the arbitration function of conflicting information objects may apply one or more customizable production rules for evaluating logical conditions defined for morphological, lexical, syntactic, semantic and (or) other attributes of the extracted information objects. In an illustrative implementation example, the attributes of the extracted information objects evaluated by production rules may include confidence levels associated with the corresponding information objects. Confidence levels can be created by the information extraction methods used in steps 120-130.

[00061] Alternatively, the arbitration function of conflicting entities may include one or more machine learning classifiers. In an illustrative example, the classifier used by the “arbitration” function of conflicting objects can provide the degree of validity of one or more possible results of the “arbitration” (for example, merging possible conflicting objects into a single object, confirming one or more extracted objects, changing one or more extracted objects and (or) deletion of one or more extracted objects). In another illustrative example, the classifier used by the “arbitration” function of conflicting objects can record the similarity of two or more information objects representing the same real-life object. If the probability exceeds a threshold value, the function of “arbitration” of conflicting objects can merge the identified information objects. In yet another illustrative example, a classifier used by the arbitrage function of conflicting objects can create degrees of confidence associated with each information object from a plurality of conflicting information objects. The arbitration function of conflicting objects can delete one or more information objects with a minimum degree of certainty and (or) a degree of certainty below a given threshold.

[00062] Classifiers used by the function of "arbitration" of conflicting objects can be trained based on pre-existing or dynamically created training data samples, based on which the results of the "arbitration" can be correlated with morphological, lexical, syntactic, semantic and (or) other attributes of extracted information objects. In certain implementations, the attributes of the extracted information objects evaluated by classifiers based on machine learning may include confidence levels associated with the corresponding information objects. Classification methods may include differential evolution methods, genetic algorithms, random forest methods, neural networks, etc.

[00063] The classifier can be trained using pre-existing and (or) dynamically created training data samples, based on which the possible results of the “arbitration” are correlated with lexical, syntactic, semantic and (or) other attributes of the text in natural language. A training data set may include one or more natural language texts accompanied by metadata that defines the sets of conflicting information objects, their classification, attributes and the corresponding results of the “arbitration” of conflicting objects. In an illustrative example, metadata may be represented by markup associated with natural language text. In certain implementations, the training data set can be iteratively improved by adding new natural language texts, accompanied by user-confirmed metadata. Validation of metadata may include receiving user input through a graphical user interface that confirms or corrects the extracted information objects and their attributes. In some implementations, user-verified texts and their accompanying metadata specifying the extracted information objects, their attributes and text annotations can be used to generate and update training samples used to train classifiers as described in detail below.

[00064] With reference to FIG. 1, in step 170, the computing system can use the final list of information objects to perform tasks or natural language processing operations, including machine translation, semantic indexing, semantic search (including semantic search in several languages), document classification, search and presentation of electronic documents (e-discovery) etc. In some implementations, the computer system may display the extracted information objects with visual links to fragments of text in natural language and take into account user actions that confirm or modify text annotations and / or the result of the classification of the object.

[00065] In some embodiments, the definition of the classifier used to extract the information may include one or more parameters, the values of which can be adjusted using machine learning methods, and may also include one or more hyperparameters, the values of which can be determined by some external to machine learning methods by operations or processes. In other words, the values of the classifier hyperparameters are determined before the application of machine learning methods for fine-tuning the classifier parameters.

[00066] In one illustrative example, the hyperparameter of the information extraction classifier can be represented by the regularization parameter of the gradient boosting classifier. In another illustrative example, the hyperparameter of the information extraction classifier can be represented by the parameter β of the F-measure, obtained by evaluating the information object issued by the classifier, which is determined as follows:

[00067]

where Accuracy = t _p / (t _p + f _p ) and Completeness = t _p / (t _p + f _n ),

t _p is the number of truly positive results (correctly classified extracted information objects), f _p is the number of false positive results (an information object that does not belong to a particular class was classified as belonging to this class), af _n is the number of false negative results (an information object that belongs to a particular class has not been classified as belonging to this class).

[00068] In some embodiments, the values of one or more hyperparameters of the information extraction classifier can be iteratively controlled by iteratively performing the operations of training the information extraction classifier, training the conflict arbitration classifier, performing the conflict arbitration function, and modifying the values of the hyper parameters based on the result of applying the function " arbitration ”of conflicts, thus iterative optimization of the quality indicator is made to evaluate the final list of involved information objects, as will be described in more detail later in this document with reference to FIG. 3.

[00069] In FIG. 3 is a flow chart of one example of a classifier training method used to extract information from natural language texts in accordance with one or more embodiments of the present invention. The method 300 and / or each of its individual functions, programs, subprograms, or operations can be performed by one or more processors of a computing system (eg, computing system 1000 in FIG. 16) that implement this method. In some embodiments, method 300 may be performed in a single processing stream. In an alternative approach, method 300 may be implemented by two or more processing threads, with one or more separate functions, procedures, routines, or operations of this method being implemented in each thread. In an illustrative example, processing streams that implement method 300 can be synchronized (for example, using semaphores, critical sections, and / or other thread synchronization mechanisms). In an alternative approach, processing streams implementing method 300 may be performed asynchronously with respect to each other.

[00070] At step 310, the computing system implements a method for training a classifier for extracting information to extract multiple information objects from a text in a natural language. In some embodiments, the definition of a classifier may include one or more parameters, the values of which are adjusted during the training operation of the classifier, and may additionally include one or more hyperparameters, the values of which can be determined by an operation or process external to machine learning. In one illustrative example, the hyperparameter of the information extraction classifier can be represented by the regularization parameter of the gradient boosting classifier. In another illustrative example, the hyperparameter of the information extraction classifier can be represented by the parameter β of the F-measure, obtained by evaluating the information object issued by the classifier.

[00071] In one illustrative example, the annotated body of texts used to train a classifier for extracting information can be divided into a training data set and a set of test data. The training data sample can then be used to determine the values of the classifier parameters, and the test data set can be used to calculate the quality of information extraction (that is, be represented by a parameterized F-measure). In one illustrative example, the learning process may include setting up one or more parameters of the information extraction classifier until the selected quality indicator applied to the extracted information objects corresponds to a predetermined threshold value.

[00072] Separation of a text corpus may include cross-validation of a training data set and a set of verification data. To reduce the variability of the result, several cross-validation iterations can be performed using various divisions, and the model verification results can be aggregated (for example, averaged) over iterations. In an illustrative example, a k-fold cross-validation method may be applied to the body of natural language texts. The method may include randomly dividing the original text corpus into k data sets of the same size, one of which is then used as a set of verification data, and the remaining k-1 additional sets are then used as training data samples. The cross-validation process can then be repeated k times so that each of the k data sets is used once as validation data. After that, k results can be aggregated to create a single calculation.

[00073] In step 320, the computing system may use a classifier for extracting information to a set of verification data, thereby obtaining the first set of information objects.

[00074] In step 330, the computing system can apply a second information extraction method to the same set of test data that is different from the information extraction classifier mentioned above, thereby obtaining a second set of information objects. In one illustrative example, the second method of extracting information can use a set of customizable production rules to interpret a set of semantic-syntactic structures obtained by semantic-syntactic analysis of text in natural language, as described in more detail later in this document. In another illustrative example, the second information extraction method may use one or more machine learning classifiers that are different from the above information extraction classifier, which was used to extract the first set of information objects.

[00075] At step 340, the computing system can determine one or more sets of conflicting information objects, each set including two or more information objects, at least two of which have been extracted by various methods of extracting information. In an illustrative example, a plurality of conflicting information objects may include two information objects, wherein the first information object relates to the first set of information objects extracted using the information extraction classifier, and the second information object relates to the second set of information objects extracted using another information extraction method, as discussed in more detail above.

[00076] In step 350, the computing system can apply a conflict arbitration function to each detected plurality of conflicting information objects to resolve conflicts, thereby creating a final list of information objects extracted from natural language texts, as described in more detail above.

[00077] In an illustrative example, the function of “arbitration” of conflicting objects can be implemented by a machine learning classifier trained to produce the degree of “validity” of one or more possible results of “arbitration” (for example, merging possible conflicting objects into a single object, confirming one or more extracted objects, changing one or more extracted objects and (or) deleting one or more extracted objects). In another illustrative example, the classifier used by the “arbitration” function of conflicting objects can ensure the similarity of two or more information objects representing the same real-life object. If the probability exceeds a threshold value, the function of “arbitration” of conflicting objects can merge the identified information objects. In another illustrative example, the classifier used by the “arbitration” function of conflicting objects can create confidence levels associated with each information object from the set of conflicting information objects. The arbitration function of conflicting objects can delete one or more information objects with a minimum degree of certainty and (or) a degree of certainty below a given threshold.

[00078] Alternatively, the conflict arbitration function may be implemented by one or more custom rules evaluating logical conditions defined on morphological, lexical, syntactic, semantic and / or other attributes of the extracted information objects, as described in more detail here.

[00079] In step 360, the computing system can evaluate a predetermined quality metric applied to the final set of extracted information objects. In one illustrative example, a quality score is provided using an F-measure.

[00080] In response to the determination in step 370 that the predetermined termination conditions are not met, processing continues at step 380 by changing one or more hyperparameters of the information extraction classifier. In one illustrative example, the hyperparameter can be represented by the β parameter of the F-measure, which is used to evaluate the quality of information extraction. Thus, the computing system can modify the value of the parameter β to increase the accuracy or completeness of the results obtained from the information extraction classifier. In some embodiments, the computing system may search a grid of parameters by iteratively changing the values of one or more hyperparameters to predetermined increments so that the parameters remain within certain ranges, thereby optimizing the quality score used for the final set of extracted information objects. After performing operations in step 380, the method may return to step 310.

[00081] In response to the determination at step 370 of the fact that the termination conditions are met, the method may exit the loop and continue to operation at step 390. In one illustrative example, the termination conditions can be determined by the maximum number of iterations performed. In response to the determination of the fact that the specified number of operations has been performed, the computing system can, at step 390, identify among the values indicated above the predetermined quality indicators evaluated at step 370 that were created at all iterations performed, the optimal (i.e., minimum or maximum) value of the quality indicator and appropriately identify the values of the hyperparameters of the information extraction classifier, which was used in the iteration, which showed the optimal quality indicator.

[00082] In another illustrative example, the termination conditions checked in step 370 can determine a threshold value corresponding to the quality indicator that was evaluated in step 370. In response to determining whether the quality indicator matches the specified threshold value, the computing system can detect hyperparameter values in step 390 information extraction classifier that was used in the last iteration that yielded a quality indicator value corresponding to the termination conditions tions. In response to determining the values of the hyperparameters of the information extraction classifier, the method may stop working. The trained information extraction classifier can then be used in the information extraction method 100.

[00083] As noted above, in some implementations, a conflict arbitration function may be performed by a machine learning classifier. The definition of such a classifier may include one or more parameters, the values of which can be adjusted using machine learning methods and may additionally include one or more hyperparameters, the values of which can be determined by some external to the machine learning operations or process. In other words, the values of the classifier hyperparameters are determined before the application of machine learning methods to fine tune the parameters of the classifier.

[00084] In an illustrative example, the conflict arbitration classifier hyperparameter may be represented by a gradient boost classifier regularization parameter. In another illustrative example, the hyperparameter of the arbitration classifier can be represented by the parameter β of the F-measure indicator by evaluating the information objects obtained by the classifier.

[00085] In some implementations, the values of one or more hyperparameters of a conflict arbitration classifier can be iteratively adjusted by iteratively performing operations to extract the first set of information objects, extract the second set of information objects, change hyper parameter values based on the quality of information extraction, and train the conflict arbitrator classifier , thus iteratively optimizing the quality metric, evaluating the final list of extracted information tion objects. In some implementations, a computer system can implement a grid search by iteratively changing the values of one or more hyperparameters using predetermined increments, while the parameters remain in certain ranges, thereby optimizing the quality metric applied to the final set extracted information objects, as described in more detail below with reference to FIG. four.

[00086] In FIG. 4 is a flowchart of another example of a classifier training method used to extract information from natural language texts in accordance with one or more aspects of the present disclosure. Method 400 and / or each of its individual functions, subprograms, subprograms, or operations may be performed by one or more processors of a computer system (eg, computer system 1000 in FIG. 16) that implements this method. In some implementations, method 400 may be performed by a single processing thread. Alternatively, method 400 may be performed by two or more processing threads, with each thread implementing one or more separate functions, routines, routines, or operations of this method. In an illustrative example, a method for implementing 400 processing threads may be synchronized (for example, using semaphores, critical sections, and / or other thread synchronization mechanisms). Alternatively, the method 400 for implementing processing threads may be performed asynchronously with respect to each other.

[00087] At step 410, a computer system implementing the method can apply the first information extraction method to natural language text. In some implementations, the first method of extracting information may use custom rule sets, automatic classification methods (also known as “machine learning classifiers”), heuristic approaches and / or combinations thereof. Applying the first method of extracting information to a text in a natural language can create the first set of information objects of one or more types of information objects.

[00088] At step 420, the computer system may apply a second method of extracting information to a natural language text that is different from the first method of extracting information. In some implementations, the second method of extracting information may use custom rule sets, automatic classification methods (also known as “machine learning classifiers”), heuristic approaches and / or combinations thereof. Applying the second method of extracting information to a text in a natural language can create a second set of information objects that can differ from the first set of information objects created using the first method of extracting information.

[00089] At step 430, the computer system can change the values of one or more hyperparameters of the arbitration conflict classifier. In an illustrative example, a hyperparameter can be represented by a regularization parameter of the gradient boosting method. In another illustrative example, the hyperparameter can be represented by the parameter β of the metric of the F-measure by evaluating the information objects obtained by the classifier. In some implementations, a computer system can implement a grid search by iteratively changing the values of one or more hyperparameters using predetermined increments, while the parameters remain in certain ranges, thereby optimizing the quality metric applied to the final set extracted information objects.

[00090] At step 440, a computer system can train a conflict arbitration classifier to perform a conflict arbitration function with respect to a set of conflicting information objects to compile a final list of extracted information objects. In an illustrative example, a set of conflicting information objects may contain two information objects, so that the first information object belongs to the first set of information objects that was extracted by the information extraction classifier, while the second information object belongs to the second set of information objects that were extracted using the second extraction method information as described in more detail above.

[00091] In an illustrative example, the classifier used by the conflict arbitration function can provide the degree of validity of one or more possible arbitration results (for example, combining two or more potentially conflicting objects into one object, confirming one or more extracted objects, changing one or more extracted objects and / or deletion of one or more extracted objects). In another illustrative example, the classifier used by the conflict arbitration function may evaluate the likelihood that two or more information objects represent the same real-life object. If the probability exceeds a threshold value, the conflict arbitration function may combine the identified information objects. In another illustrative example, the classifier used by the conflict arbitration function can produce degrees of confidence associated with each information object from a plurality of conflicting information objects. The conflict arbitration function may remove one or more information objects having the smallest degrees of certainty and / or degrees of certainty that are below a predetermined threshold.

[00092] In some implementations, the definition of a conflict arbitration classifier may include one or more parameters, the values of which are set using the classifier training procedure, and may additionally include one or more hyperparameters, the values of which can be determined by some external process of machine learning or operation.

[00093] Classification methods may include differential evolution methods, genetic algorithms, random forest methods, neural networks, etc. In an illustrative example, the learning process may include setting up one or more parameters of the information extraction classifier until a predetermined threshold of the selected quality metric is reached.

[00094] The classifiers used in the conflict arbitration function can be trained using pre-existing and / or dynamically created training datasets that correlate the results of the arbitration with morphological, lexical, syntactic, semantic and / or other attributes of the extracted information objects,

[00095] A training data set may contain one or more natural language texts, accompanied by metadata that defines a lot of conflicting information objects, their classification, attributes, and corresponding arbitration results. In an illustrative example, metadata may be represented by markup associated with natural language text. In some implementations, the training data set may be iteratively expanded by adding new natural language texts, accompanied by user-verified metadata. Validation of metadata may include receiving, through a graphical user interface (GUI), user input that confirms or corrects the extracted information objects and their attributes.

[00096] In an illustrative example, the annotated text corpus used to train a conflict arbitration classifier can be divided into a training data set and a verification data set. Then, the training data set can be used to determine the values of the classifier parameters, while the test data set can be used to calculate the metrics of the quality of information extraction (for example, represented by a parameterized F-measure). In an illustrative example, the learning process may include adjusting one or more parameters of a conflict arbitration classifier until the selected quality metric applied to the final list of extracted information objects matches a predetermined threshold.

[00097] At 450, a computer system can apply a conflict arbitration function for each set of conflicting information objects detected to resolve conflicts, thus creating a final list of information objects extracted from natural language texts, as described in more detail above.

[00098] At step 460, the computer system can calculate a predefined quality metric for the final set of extracted information objects. In an illustrative example, a quality metric can be represented by an F-measure.

[00099] In response to the determination in step 470 that the predetermined termination condition is not satisfied, the method may return to block 410; otherwise, the method may exit the loop and continue processing at block 480. In an illustrative example, the termination condition may indicate the maximum number of iterations to be performed. In response to the specified number of iterations, the computer system can, at 470, identify among the values of the above-described predetermined quality metrics calculated at 460 that were obtained at all iterations performed, the optimal (e.g., maximum or minimum) value of the quality metric and, accordingly, can identify those values of the conflict arbitration classifier hyperparameters used in the iteration process that gave the optimal quality metric.

[000100] In another illustrative example, a termination condition that is evaluated in block 470 may indicate a threshold value that must be satisfied by a quality metric that was calculated in step 460. In response to determining that the quality metric corresponds to the specified threshold value, a computer system can at step 490 identify the values of the hyperparameters of the information extraction classifier that were used at the last iteration that gave a quality indicator that satisfies the condition completion. In response to identifying the values of the hyperparameters of the conflict arbitration classifier, the method may end. Then, the trained conflict arbitration classifier can be used by method 100 to extract information.

[000101] As indicated above, production rules, classifiers and (or) other methods for extracting information and "arbitration" of conflicting objects can analyze various morphological, lexical, syntactic, semantic and (or) other attributes of the text in natural language. Such attributes can be created by performing semantic-syntactic analysis of natural language text, which is schematically shown in FIG. 5.

[000102] In FIG. 5 is a flow chart of one illustrative example implementation of a method 200 for performing semantic-syntactic analysis of sentences in natural language 212 in accordance with one or more aspects of the present invention. Method 400 can be applied to one or more syntactic units (eg, sentences) included in a particular text corpus to form a plurality of semantic-syntactic trees corresponding to syntactic units. In various illustrative examples, natural language sentences to be processed by method 400 can be extracted from one or more electronic documents that can be generated by scanning (or other means of obtaining images of paper documents) and optical character recognition (OCR) to obtain texts corresponding to these documents. Natural language sentences can also be extracted from various other sources, including e-mail messages, texts from social networks, digital content files processed using speech recognition methods, etc.

[000103] In block 214, a computing device implementing the method may conduct a lexical and morphological analysis of sentence 212 to establish the morphological meanings of words included in the sentence. As used herein, the "morphological meaning" of a word means one or more lemmas (i.e., canonical or dictionary forms) corresponding to a word, and a corresponding set of grammatical attribute values that define the grammatical meaning of the word. Such grammatical features may include the lexical category (part of speech) of the word and one or more morphological and grammatical features (for example, case, gender, number, conjugation, etc.). Due to the homonymy and (or) matching grammatical forms corresponding to different lexical and morphological meanings of a particular word, two or more morphological meanings can be established for a given word. A more detailed description of an illustrative example of the lexical-morphological analysis of sentences is described in more detail below in this document with reference to FIG. 6.

[000104] In block 215, the computing device may conduct coarse syntax analysis of sentence 212. Coarse syntax analysis may include the use of one or more syntactic models, which can be correlated with elements of sentence 212, with the subsequent establishment of surface (ie syntactic) relationships in the framework of Proposition 212 to obtain a graph of generalized components. As used herein, “constituent” means a group of neighboring words of an original sentence that functions as one grammatical entity. A component includes a core in the form of one or more words and may also include one or more child components at lower levels. A child component is a dependent component that can be correlated with one or more parent components.

[000105] At block 216, the computing device can parse sentence 212 accurately to generate one or more sentence syntax trees. Among the various syntax trees, based on a specific evaluation function, taking into account the compatibility of the lexical meanings of the words of the original sentence, surface relations, deep relations, etc. one or more of the best syntax trees matching Proposition 212 can be selected.

[000106] In block 217, the computing device can process the syntax trees to form a semantic structure 218 corresponding to Proposition 212. The semantic structure 218 may include many nodes corresponding to semantic classes and may also include many arcs corresponding to semantic relations (for a more detailed description see below. in this document).

[000107] FIG. 6 schematically illustrates an example of the lexical-morphological structure of a sentence in accordance with one or more aspects of the present invention. An example of lexical-morphological structure 300 may include many pairs of "lexical meaning - grammatical meaning" for an example sentence. As an illustrative example, "ll" can be correlated with the lexical meaning "shall" 512 and "will" 514. The grammatical meaning associated with the lexical meaning 512 is <Verb, GTVerbModal, ZeroType, Present, Nonnegative, Composite II>. The grammatical meaning associated with the lexical meaning 314 is <Verb, GTVerbModal, ZeroType, Present, Nonnegative, Irregular, Composite II>.

[000108] FIG. 7 schematically illustrates language descriptions 210 used, including morphological descriptions 201, lexical descriptions 203, syntactic descriptions 202, and semantic descriptions 204, as well as relationships between them. Among them, morphological descriptions 201, lexical descriptions 203, and syntactic descriptions 202 are language dependent. The set of language descriptions 610 is a model of a particular natural language.

[000109] As an illustrative example, a specific lexical meaning in lexical descriptions 203 may be correlated with one or more surface models of syntactic descriptions 202 corresponding to a given lexical meaning. A specific surface model of syntactic descriptions 202 can be correlated with an in-depth model of semantic descriptions 204.

[000110] In FIG. 8 schematically illustrates several examples of morphological descriptions. The components of morphological descriptions 201 may include: descriptions of inflection 710, grammar system 720, descriptions of derivation 730 and others. Grammar system 720 includes a set of grammatical categories, such as a part of speech, case, gender, number, person, recurrence, time, type and their meanings (so-called "grammes"), including, for example, an adjective, noun or verb; nominative, accusative or genitive; feminine, masculine or neuter, etc. Corresponding grammars can be used to compose a description of inflection 710 and a description of derivation 730.

[000111] The description of the inflection 710 defines the forms of a given word depending on its grammatical categories (for example, case, gender, number, time, etc.) and in the broad sense includes or describes various possible forms of the word. Description of word formation 730 determines which new words can be formed from a given word (for example, compound words).

[000112] In accordance with one aspect of the present invention, component models can be used to establish syntactic relationships between elements of the original sentence. A component is a group of neighboring words in a sentence that behave as a whole. The core component is the word; it can also contain child components of lower levels. The child component is a dependent component and can be attached to other components (parent) to build syntactic descriptions 202 of the original sentence.

[000113] In FIG. Figure 9 shows examples of syntactic descriptions. The components of syntax descriptions 202 may include, but are not limited to, surface models 910, descriptions of surface positions 920, descriptions of referential and structural controls 956, descriptions of control and alignment 940, descriptions of non-wood syntax 950, and analysis rules 960. Syntax descriptions 202 can be used to build possible syntactic structures of the original sentence in a given natural language, taking into account the free linear word order, non-wood syntactic phenomena (for example, matching , ellipsis, etc.), referential relations and other factors.

[000114] Surface models 910 can be represented as sets of one or more syntactic forms (“synths” 912) to describe possible syntactic sentence structures that are part of syntax description 202. In general, the lexical meaning of a word in a natural language can be associated with surface (syntactic) models 910. The surface model can represent the components that are possible if the lexical meaning acts as a "core". A surface model may include a set of surface positions of child elements, a description of the linear order, and / or diathesis. As used herein, “diathesis” means a definite relationship between surface and depth positions and their semantic roles expressed through depth positions. For example, diathesis can be expressed by the pledge of the verb: if the subject is the agent of the action, the verb is the active pledge, and when the subject is the direction of the action, this is expressed by the passive pledge of the verb.

[000115] In the component model, a plurality of surface positions 915 of the daughter components and descriptions of their linear order 916 can be used to describe grammatical values of 914 possible placeholders for these surface positions. Diathesis 917 represents the correspondence between the surface positions 915 and the deep positions 514 (as shown in Fig. 11). Communicative descriptions 980 describe the communicative order in a sentence.

[000116] A description of the linear order (916) can be presented in the form of linear order expressions reflecting the sequence in which various surface positions (915) can occur in the sentence. The expressions of linear order may include the names of variables, the names of surface positions, parentheses, grammes, the operator "or" (or), etc. As an illustrative example, the linear ordering description for the Boys play football simple sentence can be represented as Subject Core Object_Direct, where Subject, Core, Object_Direct are names of surface positions 915 corresponding to the word order.

[000117] Communicative descriptions 980 can describe the word order in synthform 912 from the point of view of communicative acts represented as communicative order expressions that are similar to linear order expressions. Control and negotiation descriptions 940 may include rules and grammar constraints on the attachments that are used during parsing.

[000118] Descriptions of non-wood syntax 950 can be created to reflect various linguistic phenomena, such as ellipsis and matching, they are used in transformations of syntactic structures that are created at various stages of analysis in various embodiments of the invention. Descriptions of non-wood syntax 950 may include, but are not limited to, descriptions of ellipsis 952, descriptions of alignment 954, and descriptions of referential and structural controls 930.

[000119] Analysis Rules 960 may describe the properties of a particular language and be used as part of semantic analysis. The analysis rules 960 can include the rules for calculating semantems 962 and the normalization rules 964. The normalization rules 964 can be used to describe transformations of semantic structures that may differ in different languages.

[000120] In FIG. 10 gives an example of semantic descriptions. The components of semantic descriptions 204 are language independent and may, among other things, include a semantic hierarchy 510, descriptions of deep positions 520, a semantem system 530, and pragmatic descriptions 540.

[000121] The core of semantic descriptions can be represented by a semantic hierarchy 510, which can include semantic concepts (semantic entities), also called semantic classes. The latter can be arranged in a hierarchical structure that reflects the parent-child relationship. In general, a child semantic class can inherit one or more properties of its direct parent and other semantic ancestor classes. As an illustrative example, the semantic class SUBSTANCE (Substance) is a child semantic class of the ENTITY class and the parent semantic class for the classes GAS, (Gas), LIQUID (Liquid), METAL (Metal), WOOD_MATERIAL (Wood), etc.

[000122] Each semantic class in the semantic hierarchy 510 may be accompanied by a depth model 512. The depth model 512 of the semantic class may include many depth positions 514, which may reflect the semantic roles of child components in various sentences with objects of a given semantic class as the core of the parent component. The depth model 512 may also include possible semantic classes that act as placeholders for deep positions. Deep positions (514) can express semantic relationships, including, for example, "agent" (agent), "addressee" (destination), "instrument" (instrument), "quantity" (quantity), etc. A child semantic class can inherit and refine the deep model of its immediate parent semantic class.

[000123] The descriptions of the deep positions 520 reflect the semantic roles of the child components in the deep models 512 and can be used to describe the general properties of the deep positions 514. The descriptions of the deep positions 520 may also contain grammatical and semantic restrictions on the placeholders of the deep positions 514. Properties and restrictions associated with with depth positions 514 and their possible placeholders in various languages, can be largely similar and often identical. Thus, the deep positions 514 are language independent.

[000124] The semantem system 530 may be a plurality of semantic categories and semantems that represent semantic category meanings. As an illustrative example, the semantic category "DegreeOfComparison" (Comparison Degree) can be used to describe the degree of comparison of adjectives and include the following semantems: "Positive", "ComparativeHigherDegree" (Comparative Comparison), "SuperlativeHighestDegree" (Superlative Comparison) and others. As another illustrative example, the semantic category "RelationToReferencePoint" (Relation to a point) can be used to describe the order (spatial or temporal in the broad sense of the analyzed words), such as before or after a point or event, and include semantems "Previous" ( Previous) and Subsequent. As another illustrative example, the semantic category "EvaluationObjective" can be used to describe an objective evaluation, such as, for example, "Bad", "Good", etc.

[000125] The semantem system 530 may include language-independent semantic attributes that can express not only semantic characteristics, but also stylistic, pragmatic, and communicative characteristics. Some semanthemes can be used to express an atomic meaning, which finds a regular grammatical and (or) lexical expression in natural language. According to their intended purpose and use of the system, semantems can be divided into categories, for example, grammatical semantems 532, lexical semantems 534 and classifying grammatical (differentiating) semantems 536.

[000126] Grammatical semantems 532 can be used to describe the grammatical properties of constituents when transforming a syntax tree into a semantic structure. The lexical semantems 534 can describe specific properties of objects (for example, “being flat” or “being liquid”) and used in descriptions of deep positions 520 as a restriction of placeholders for deep positions (for example, for the verbs “face (with ) "(facing) and" flood "(fill), respectively). Classifying grammatical (differentiating) semantems 536 can express the differential properties of objects within a single semantic class. As an illustrative example, in the semantic class HAIRDRESSER (HAIRDRESSER), the semantema <<RelatedToMen>> (applies to men) is assigned the lexical meaning "barber" in contrast to other lexical values that also belong to this class, for example, "hairdresser", "hairstylist " etc. Using data language-independent semantic properties that can be expressed as elements of a semantic description, including semantic classes, deep positions, and semantems, semantic information can be extracted in accordance with one or more aspects of the present invention.

[000127] The pragmatic descriptions 540 allow you to assign a specific theme, style or genre to the texts and objects of the semantic hierarchy 510 (for example, “Economic Policy”, “Foreign Policy”, “Jurisprudence”, “Legislation”, “Trade”, “Finance”, etc. .d.). Pragmatic properties can also be expressed by semantems. As an illustrative example, pragmatic context can be taken into account in semantic analysis.

[000128] In FIG. 11 is an example of lexical descriptions. Lexical descriptions (203) are the set of lexical meanings 612 of a particular natural language. For each lexical meaning 612, there is a relationship 602 with its language-independent semantic parent in order to indicate the position of any given lexical meaning in the semantic hierarchy 510.

[000129] The lexical meaning 612 in the lexical-semantic hierarchy 510 can be correlated with the surface model 910, which in turn can be correlated through one or more diathesis 917 with the corresponding depth model 512. The lexical value 612 can inherit the semantic class of its parent and specify its deep model 512.

[000130] The lexical surface model 910 may include one or more syntactic forms 912. The surface model 910 synth 912 may include one or more surface positions 915, including corresponding descriptions of their linear order 916, one or more grammatical values 914, expressed in in the form of a set of grammatical categories (grammes), one or more semantic restrictions associated with placeholder surface positions, and one or more diathesis 917. Semantic restrictions related Items with a specific placeholder for a surface position can be represented as one or more semantic classes whose objects can fill this surface position.

[000131] In FIG. 12 schematically illustrates examples of data structures that can be used within the framework of one or more methods of the present invention. Referring again to FIG. 5, in block 214, a computing device that implements this method can conduct a lexical-morphological analysis of sentence 212 to construct a lexical-morphological structure 722 according to FIG. 12. The lexical-morphological structure 722 may include many correspondences of lexical and grammatical meanings for each lexical unit (for example, a word) of the original sentence. FIG. 6 schematically illustrates an example of a lexical-morphological structure.

[000132] Returning again to FIG. 6, at block 215, the computing device may conduct a rough parsing of the original sentence 212 to construct a graph of generalized components 732 according to FIG. 12. Rough parsing involves applying one or more possible syntactic models of possible lexical meanings to each element of the set of elements of the lexical-morphological structure 722, in order to establish the set of potential syntactic relations in the initial sentence 212, represented by the graph of generalized components 732.

[000133] The graph of generalized components 732 can be represented by an acyclic graph that includes many nodes corresponding to the generalized components of the original sentence 212 and includes many arcs corresponding to surface (syntactic) positions that can express different types of relations between generalized lexical values. In the framework of this method, many potentially applicable syntactic models can be applied for each element of the set of elements of lexical and morphological structures of the original sentence 212 to form a set of components of the initial sentence 212. Then, within the framework of the method, many possible components of the initial sentence 212 can be considered for constructing a graph of generalized components 732 based on a set of components. The graph of generalized components 732 at the level of the surface model can reflect many potential connections between the words of the original sentence 212. Since the number of possible syntactic structures can be relatively large, the graph of generalized components 732 can, in general, include redundant information, including a relatively large number of lexical values at certain nodes and (or) surface positions along certain arcs of the graph.

[000134] The graph of generalized components 732 can initially be constructed in the form of a tree, starting from the end nodes (leaves) and moving further to the root by adding child components that fill the surface positions 915 of the set of parent components so that all lexical units of the original sentence are covered 212.

[000135] In some embodiments, the root of the graph of generalized components 732 is a predicate. In the process described above, a tree can become a graph, as certain lower-level components can be included in one or more upper-level components. Many components that represent certain elements of the lexical-morphological structure can then be generalized to obtain generalized components. Components can be generalized based on their lexical meanings or grammatical meanings 914, for example, on the basis of parts of speech and relations between them. FIG. 13 schematically illustrates an example of a graph of generalized components.

[000136] At block 216, the computing device can parse sentence 212 to form one or more syntax trees 742 of FIG. 12 based on the graph of generalized components 732. For each syntax tree, the computing device can determine the integral estimate based on a priori and calculated estimates. The tree with the best rating can be selected to build the best syntactic structure 746 of the original sentence 212.

[000137] During the construction of the syntax structure 746 based on the selected syntax tree, the computing device can establish one or more non-wood links (for example, by creating an additional link among at least two nodes of the graph). If this process fails, the computing device can select a syntax tree with a conditionally optimal estimate that is closest to the optimal one, and an attempt is made to establish one or more non-wood links in the tree. Finally, as a result of accurate parsing, a syntax structure 746 is created that represents the best syntax structure corresponding to the original sentence 212. In fact, by selecting the best syntax structure 746, the best lexical values 240 for the elements of the original sentence 212 are determined.

[000138] At block 217, the computing device can process the syntax trees to form a semantic structure 218 corresponding to sentence 212. The semantic structure 218 can reflect semantics transmitted by the original sentence in language-independent terms. The semantic structure 218 can be represented in the form of an acyclic graph (for example, a tree, possibly supplemented by one or more non-wood links (the arc of the graph). The words of the original sentence are represented by nodes with the corresponding language-independent semantic classes of the semantic hierarchy 510. The arcs of the graph represent deep (semantic ) the relationship between the elements of the sentence. The transition to the semantic structure 218 can be carried out using the rules of analysis 960 and involves the correlation of one or more attributes ( reflecting the lexical, syntactic and (or) semantic properties of the words of the original sentence 212) with each semantic class.

[000139] In FIG. 14 is an example of a syntactic structure of a sentence generated from a graph of generalized components shown in FIG. 13. Node 901 corresponds to the lexical element “life” 906. Using the method of syntactic-semantic analysis described in this document, the computing device can determine that the lexical element “life” 906 represents one form of lexical meaning correlated with with the semantic class "LIVE" 904 and fills the surface position $ Adjunct_Locative 905) in the parent component represented by the Verb: succeed: succeed: TO_SUCCEED control node (907).

[000140] In FIG. 15 shows a semantic structure corresponding to the syntax structure in FIG. 15. With respect to the aforementioned lexical element “life” (906) in FIG. 13, the semantic structure includes the lexical class 1010 and the semantic class 1030, corresponding to those presented in FIG. 15, however, instead of the surface position (905), the semantic structure includes the depth position "Sphere" (scope_of_operation) 1020.

[000141] As noted above in this document, a “ontology” can be a model, which is an object belonging to a certain field of knowledge (subject area), and the relationship between these objects. Thus, the ontology differs from the semantic hierarchy, despite the fact that it can be correlated with the elements of the semantic hierarchy through certain relationships (also called “anchors”). An ontology may include definitions of a certain set of classes, where each class corresponds to the concept of a subject area. Each class definition may include definitions of one or more objects assigned to that class. According to generally accepted terminology, an ontology class may also be called a “concept”, and an object belonging to the class may mean an instance of this concept.

[000142] In accordance with one or more aspects of the present invention, a computing device that implements the methods described herein may index one or more parameters resulting from semantic-syntactic analysis. Thus, the methods of the present invention allow us to consider not only many words in the composition of the original text corpus, but also many lexical meanings of these words, storing and indexing all the syntactic and semantic information obtained in the course of parsing and semantic analysis of each sentence of the original text corpus. Such information may additionally include data obtained during the intermediate stages of the analysis, as well as results of lexical selection, including results obtained during the resolution of ambiguities caused by homonymy and (or) matching grammatical forms corresponding to different lexical and morphological meanings of some words of the original language.

[000143] For each semantic structure, one or more indexes can be created. An index can be represented as a data structure in memory, for example, in the form of a table consisting of several records. Each record can be a correspondence between a particular element of the semantic structure (for example, one word or several words, syntactic relation, morphological, syntactic or semantic property or syntactic or semantic structure) and one or more identifiers (or addresses) of the use of this element of semantic structures in the source text.

[000144] In some embodiments, the implementation of the index may include one or more values of morphological, syntactic, lexical and (or) semantic parameters. These values can be created during a two-stage semantic analysis (for a more detailed description, see this document). The index can be used to perform various natural language processing tasks, including semantic search.

[000145] A computing device that implements this method can extract a wide range of lexical, grammatical, syntactic, pragmatic and (or) semantic characteristics during syntactic-semantic analysis and the creation of semantic structures. In an illustrative example, the system can extract and store certain lexical information, data on the membership of certain lexical units in semantic classes, information on grammatical forms and linear order, information on the use of certain forms, aspects, tonality (for example, positive or negative), deep positions, non-wood connections, semantems, etc.

[000146] A computing device that implements the methods described herein can analyze using one or more of the text analysis methods described in this document and index any one or more parameters of language descriptions, including lexical values, semantic classes, grammes, semantems, and etc. Indexing a semantic class can be used in various natural language processing tasks, including semantic search, classification, clustering, text filtering, etc. Indexing lexical values (instead of indexing words) allows you to search not only words and word forms, but also lexical meanings, those. words with certain lexical meanings. A computing device that implements the methods of the present invention can also store and index syntactic and semantic structures created by one or more of the text analysis methods described herein to use these structures and (or) indices for semantic search, classification, clustering and filtering of documents .

[000147] In FIG. 16 shows an illustrative example of a computing system 1000 that can execute a set of instructions that cause the computing system to execute any one or more of the methods of the present invention. A computing system may be connected to another computing system via a local area network, a corporate network, an extranet, or the Internet. A computing system can operate as a server or client in a client-server network environment, or as a peer-to-peer computing device in a peer-to-peer (or distributed) network environment. A computing system can be represented by a personal computer (PC), a tablet PC, a television set-top box (STB), a pocket PC (PDA), a cell phone, or any computer system capable of executing a set of commands (sequentially or otherwise) that determine the operations that should be made by this computing system. In addition, although only one computing system is shown, the term “computing system” may also include any combination of computing systems that separately or collectively execute a set (or more sets) of instructions to execute one or more of the methods discussed herein .

[000148] An example of a computing system 1000 includes a processor 502, a main storage device 504 (eg, read only memory (ROM) or dynamic random access memory (DRAM)) and a data storage device 518 that communicate with each other via a bus 530.

[000149] The processor 502 may be represented by one or more general-purpose computing systems, for example, a microprocessor, a central processor, etc. In particular, the processor 502 may be a full instruction set microprocessor (CISC), a reduced instruction set microprocessor (RISC), an extra long instruction microprocessor (VLIW), a processor implementing another instruction set, or processors implementing a combination of instruction sets. The processor 502 may also be one or more special-purpose computing systems, for example, a custom integrated circuit (ASIC), field programmable gate array (FPGA), digital signal processor (DSP), network processor, and the like. The processor 502 is configured to execute instructions 526 to perform the operations and functions discussed herein.

[000150] Computing system 1000 may further include a network interface device 522, a visual display device 510, a character input device 512 (eg, a keyboard), and a touch screen input device 514.

[000151] The data storage device 518 may include a computer-readable storage medium 524 that stores one or more sets of instructions 526 and that implements one or more of the methods or functions discussed herein. The instructions 526 may also reside completely or at least partially in the main memory 504 and / or in the processor 502 while they are being executed in the computer system 1000, and the random access memory 504 and the processor 502 also constitute a computer-readable storage medium. Commands 526 may also be transmitted or received over network 516 through a network interface device 522.

[000152] In some embodiments of the invention, the instruction set 526 may comprise instructions of a method 100, 300, and / or 400 for extracting information from natural language texts and training classifiers in accordance with one or more embodiments of the present invention. Although the computer readable storage medium 524 is shown in the example of FIG. 16 as a single medium, the term “machine-readable medium” should be understood in a broad sense, meaning one or more media (for example, a centralized or distributed database and / or corresponding caches and servers) that store one or more sets of instructions. The term "computer-readable storage medium" should also be understood as including any medium that can store, encode or transfer a set of instructions for execution by a machine and which enables the machine to execute any one or more of the methods of the present invention. Therefore, the term “computer-readable storage medium” refers, inter alia, to solid-state storage devices, as well as to optical and magnetic media.

[000153] The methods, components, and functions described in this document can be implemented using discrete equipment components or they can be integrated into the functions of other equipment components, for example, ASICS (specialized custom integrated circuit), FPGA (programmable logic integrated circuit) , DSP (digital signal processor) or similar devices. In addition, methods, components and functions may be implemented using firmware modules or functional block diagrams of the hardware. The methods, components and functions may also be implemented using any combination of hardware and software components or solely using software.

[000154] In the above description, numerous details are set forth. However, it should be apparent to any person skilled in the art who has read this description that the present invention can be practiced without these specific details. In some cases, well-known structures and devices are shown in block diagrams without detail, so as not to complicate the description of the present invention.

[000155] Some parts of the description of preferred embodiments of the invention are presented in the form of algorithms and a symbolic representation of operations with data bits in a computer storage device. Such descriptions and representations of algorithms represent the means used by specialists in the field of data processing, which ensures the most efficient transfer of the essence of work to other specialists in this field. In the context of the present description, as is customary, an algorithm is a logically consistent sequence of operations leading to the desired result. Operations involve actions that require physical manipulation of physical quantities. Usually, although not necessarily, these quantities take the form of electrical or magnetic signals that can be stored, transmitted, combined, compared and other manipulations performed. Sometimes it is convenient, first of all for ordinary use, to describe these signals in the form of bits, values, elements, symbols, terms, numbers, etc.

[000156] However, it should be borne in mind that all these and similar terms should be associated with the corresponding physical quantities and that they are only convenient designations applicable to these quantities. Unless explicitly stated otherwise, it is assumed that in the following description the terms “determination”, “calculation”, “calculation”, “receipt”, “establishment”, “determination”, “change”, etc. relate to the actions and processes of a computing system or similar electronic computing system that uses and converts data represented as physical (e.g. electronic) quantities in registers and storage devices of a computing system into other data also represented as physical quantities in storage devices or computer system registries or other devices for storing, transmitting or displaying such information.

[000157] The present invention also relates to a device for performing the operations described herein. Such a device can be specially designed for the required purposes, or it can be a universal computer, which is selectively activated or additionally configured using a program stored in a computer storage device. Such a computer program may be stored on a computer-readable storage medium, for example, but not limited to, any type of disk, including floppy disks, optical disks, CD-ROMs and magneto-optical disks, read-only memory (ROM), random access memory (RAM), EPROM , EEPROM, magnetic or optical cards and any type of media suitable for storing electronic information.

[000158] It should be understood that the above description is intended to illustrate, and not limit the essence of the invention. Various other embodiments of the invention will become apparent to those skilled in the art after reading and understanding the above description. Based on this, the scope of the invention should be determined taking into account the attached claims, as well as all areas of application of equivalent methods, which are equally covered by the claims.

Claims (54)

1. A method of extracting information from natural language texts, including: extracting the first set of information objects from a text in a natural language by a computer system; extracting from the text in the natural language of the second set of information objects; creating an intermediate list of information objects, including at least a subset of the first set of information objects and at least a subset of the second set of information objects; identifying in the intermediate list of information objects a plurality of conflicting information objects, in which a first information object of a plurality of conflicting information objects belongs to a first plurality of information objects, and a second information object of a plurality of conflicting information objects belongs to a second plurality of information objects; and creation of a final list of information objects extracted from natural language text by applying to the set of conflicting information objects the function of “arbitration” of conflicting objects, which performs at least one of the following actions: changing the first information object, deleting the first information object, or merging two or more information objects from a variety of conflicting information objects. 2. The method according to p. 1, characterized in that the extraction of the first set of information objects is performed using the first information extraction method, and the extraction of the second set of information objects is performed using the second information extraction technique. 3. The method according to claim 1, further comprising: use of the final list of information objects to perform natural language processing operations. 4. The method according to p. 1, further comprising: Presentation of the final list of information objects using an RDF graph. 5. The method according to p. 1, characterized in that the identification of multiple conflicting information objects further includes: determining that the first text annotation associated with the first information object intersects with the second text annotation associated with the second information object. 6. The method according to p. 1, characterized in that the identification of multiple conflicting information objects further includes: evaluation of the logical condition, including the first attribute of the first information object and the second attribute of the second information object. 7. The method according to p. 1, characterized in that the identification of multiple conflicting information objects further includes: the definition of an information object having a number of attributes of a certain type in an amount above the threshold number of attributes of a certain type. 8. The method according to p. 1, characterized in that the function of "arbitration" of conflicting objects further includes: applying multiple production rules to multiple natural language text attributes. 9. The method according to p. 1, characterized in that the function of "arbitration" of conflicting objects further includes: applying to a set of attributes of a text in a natural language a machine learning classifier, based on which at least one of the following is determined: the similarity of the first information object and the second information object representing one object, the degree of confidence of the first information object, or the degree of confidence of the second information object. 10. The method according to p. 9, further comprising: acceptance of user input confirming the final list of information objects; adding to the training sample data in natural language text accompanied by metadata, including definitions and text annotations of one or more information objects, the final list of information objects; training, using a training data set, a classifier based on machine learning that implements the function of "arbitration" of conflicting objects. 11. The method according to p. 9, characterized in that the set of attributes of the text in the natural language includes one or more degrees of confidence associated with the corresponding information objects. 12. The method according to p. 1, characterized in that the extraction from the text in the natural language of the first set of information objects further includes: the application of a classifier based on machine learning to a set of attributes of a text in a natural language, thereby creating a degree of connectedness of a fragment of a text in a natural language with a given class of information objects. 13. The method according to p. 12, further comprising: the adoption of user input, confirming many information objects to create a sample of training data; application to the training sample of natural language text data accompanied by metadata, including definitions and text annotations of one or more information objects of the first set of information objects; training, using a training data set, a classifier based on machine learning used to extract information. 14. A computing system for extracting information from texts in a natural language, containing: memory; a processor associated with the specified memory, and this processor is configured to: extracting from the natural language text the first set of information objects; extracting from the text in the natural language of the second set of information objects; creating an intermediate list of information objects, including at least a subset of the first set of information objects and at least a subset of the second set of information objects; identifying in the intermediate list of information objects a plurality of conflicting information objects, in which a first information object of a plurality of conflicting information objects belongs to a first set of information objects, and a second information object of a plurality of conflicting information objects belongs to a second set of information objects; and creating a final list of information objects extracted from natural language text by applying to the set of conflicting information objects the function of “arbitration” of conflicting information objects, which performs at least one of the following actions: changing the first information object or merging two or more information objects from many conflicting information objects. 15. The computing system according to claim 14, characterized in that the extraction of the first set of information objects is performed using the first information extraction technique, and the extraction of the second set of information objects is performed using the second information extraction technique. 16. The computing system of claim 14, further comprising: use of the final list of information objects to perform natural language processing operations. 17. The computing system according to p. 14, characterized in that the function of "arbitration" of conflicting objects further includes: applying to a set of attributes of a text in a natural language a classifier based on machine learning, on the basis of which at least one of the following is determined: the similarity of the first information object and the second information object representing one object, the degree of confidence of the first information object, or the degree of confidence of the second information object. 18. A permanent computer-readable storage medium for extracting information from natural language texts, including executable instructions for a computer system, providing the ability to: extracting from the natural language text the first set of information objects; extracting from the text in the natural language of the second set of information objects; creating an intermediate list of information objects, including at least a subset of the first set of information objects and at least a subset of the second set of information objects; identifying in the intermediate list of information objects a plurality of conflicting information objects, in which a first information object of a plurality of conflicting information objects belongs to a first set of information objects, and a second information object of a plurality of conflicting information objects belongs to a second set of information objects; and creating a final list of information objects extracted from natural language text by applying to the set of conflicting information objects the function of “arbitration” of conflicting information objects, which performs at least one of the following actions: changing the first information object or merging two or more information objects from many conflicting information objects. 19. The machine-readable permanent storage medium according to claim 18, characterized in that the extraction of the first set of information objects is performed using the first information extraction method, and the extraction of the second set of information objects is performed using the second information extraction method. 20. A non-volatile computer-readable storage medium according to claim 18, further comprising: use of the final list of information objects to perform natural language processing operations.

Images