CN119066179A

CN119066179A - Question-answering processing method, computer program product, device and medium

Info

Publication number: CN119066179A
Application number: CN202411515080.4A
Authority: CN
Inventors: 范琳琳; 王武军; 徐基法
Original assignee: Suzhou Metabrain Intelligent Technology Co Ltd
Current assignee: Suzhou Metabrain Intelligent Technology Co Ltd
Priority date: 2024-10-29
Filing date: 2024-10-29
Publication date: 2024-12-03

Abstract

The application provides a question and answer processing method, a computer program product, equipment and a medium, which comprise the steps of responding to the fact that question text data are detected to be input into a processing model, determining a target first category matched with the question text data in a first vector database according to a first preset rule, determining a target second category matched with the question text data and a plurality of pending document segmentation fragments according to a second preset rule, the target first category and index information in the first vector database, performing similarity calculation on the question text data and the plurality of pending document segmentation fragments to select a first target document segmentation fragment, constructing a target retrieval result according to the first target document segmentation fragment and the question text data, and inputting the target retrieval result into the processing model to output answers matched with the question text data. By constructing the hierarchical vector database, the relevance between the text fragments obtained by retrieval and the questions is improved, and the accuracy and reliability of model answers are further improved.

Description

Question-answering processing method, computer program product, device and medium

Technical Field

The present application relates to the field of artificial intelligence technologies, and in particular, to a question-answering processing method, a computer program product, a device, and a medium.

Background

Models for question-and-answer processing are typically pre-trained on large-scale data sets to learn general knowledge to achieve dialogue through the trained models to obtain the desired relevant knowledge.

The existing model enhances the retrieval capability in the professional field by a method of externally hanging a knowledge base, but the retrieval quality is low, and the retrieval with low recall can lead to missing part of information, so that the model has inaccurate or incomplete answer to the question generation.

Therefore, a method for enhancing the model answer effect by improving the retrieval quality is needed to solve the above problems.

Disclosure of Invention

In view of the foregoing, it is desirable to provide a question-answering method, a computer program product, a device, and a medium.

In a first aspect, the present application provides a question-answering processing method, including:

In response to detecting that there is a question text data input to the processing model, determining a target first category matching the question text data in a first vector database according to a first preset rule;

determining a target second category and a plurality of pending document segmentation fragments matched with the question text data according to a second preset rule, the target first category and index information in the first vector database;

performing similarity calculation on the problem text data and the plurality of undetermined document segmentation fragments to select a first target document segmentation fragment;

and constructing a target search result according to the first target document segmentation segment and the question text data, and inputting the target search result into a processing model to output an answer matched with the question text data.

In some embodiments, the method further comprises updating the target search result, the updating method comprising:

in response to detecting that there is question text data input to a processing model, obtaining a first vector matching the question text data;

searching a second vector database for a target vector matched with the first vector;

Determining a second target document segmentation segment matched with a second vector according to the corresponding relation between the document segmentation segment and the second vector;

screening the first target document segmentation fragments and the second target document segmentation fragments to generate third target document segmentation fragments;

and updating the target retrieval result according to the third target document segmentation segment and the problem text data. The method further comprises a first vector database construction method:

Performing first classification processing on the original documents according to basic characteristics of a plurality of original documents contained in an original document set to classify the original documents in the original document set into a plurality of first categories;

Acquiring a document abstract of each first-class matched original document;

Performing second classification processing on the document summaries of the original documents in the first categories to classify the original documents in each of the first categories into a plurality of second categories;

splitting the original document in each second category to generate a plurality of first document splitting fragments;

and constructing a first vector database according to the first document segmentation fragments.

In some embodiments, the method further comprises a method of constructing a second vector database:

Acquiring all original documents contained in an original document set and segmenting the original documents to generate a plurality of second document segmentation fragments;

generating an alternative problem of matching the second document segmentation fragments;

vectorizing the alternative questions to generate the second vector, and constructing a second vector database according to the second vector.

In some embodiments, after building a first vector database from the first document segmentation segment, the method includes:

recording a first mapping relation between the first category and the second category;

Recording a second mapping relation between the second category and the first document segmentation segment;

and generating index information of the first vector database according to the first mapping relation and the second mapping relation.

In some embodiments, the first classification processing is performed on the original documents according to the basic characteristics of the plurality of original documents contained in the original document set to implement classification of the original documents in the original document set into a plurality of first categories, including:

Acquiring basic information of a plurality of original documents contained in the original document set and extracting basic features from the basic information;

basic features of a plurality of original documents contained in an original document set are input into a classification model to output a plurality of first categories matching the original document set.

In some embodiments, the classifying the document summaries of the original documents in the first categories into a plurality of second categories includes:

vectorizing the document summaries of each of the first class matches to generate summary vectors;

And performing second classification processing on the summary vectors matched with each first category according to a clustering algorithm to classify the document summaries matched with each first category into a plurality of second categories.

In some embodiments, when performing a second classification process on the summary vector matched by each first category according to a clustering algorithm to classify the document summaries matched by each first category into a plurality of second categories, the method for determining the number of the second categories under the first category includes:

selecting and calculating whether any two abstract vectors are orthogonal from the abstract vectors matched with the first category;

If two abstract vectors are orthogonal, defining an intersection point of the two abstract vectors as an initialization center point;

Repeating the step of acquiring the initialization center point until the first class matched abstract vector is traversed;

And determining the number of the second categories under the first category according to the number of the initialization center points.

In some embodiments, the determining, in the first vector database, a target first category matching the question text data according to a first preset rule includes:

Extracting question features of the question text data and inputting the question features into a classification model to output a target first class, or,

Acquiring keywords of the problem text data and pre-labeled field information;

and determining the target first category matched with the question text data by matching the keywords of the question text data with the keywords of the original document matched with each first category, matching the domain information of the question text data with the domain information of the original document matched with each first category and calculating the similarity of the question text data with the title of the original document matched with each first category.

In some embodiments, the determining, according to a second preset rule, a target first category, and index information in the first vector database, a target second category matching the question text data, and a plurality of pending document segmentation fragments includes:

confirming a plurality of second classes to be determined corresponding to the target first class according to the index information;

Calculating the Euclidean distance between the first vector corresponding to the problem text data and the average vector value of the second category to be determined;

And determining the undetermined second category corresponding to the minimum Euclidean distance as a target second category, and determining the first document segmentation fragments contained in the target second category as undetermined document segmentation fragments.

In some embodiments, the performing similarity calculation on the question text data and the plurality of pending document segmentation fragments to select a first target document segmentation fragment includes:

obtaining text vectors matched with the segmentation fragments of the undetermined document from the first vector database;

calculating the similarity between the first vector and the text vector;

selecting a preset number of target text vectors according to the ranks of the calculated similarities, and determining a pending document segmentation segment corresponding to the target text vectors as a first target document segmentation segment.

In some embodiments, the constructing the target search result according to the first target document segmentation segment and the question text data includes converting the first target document segmentation segment and the question into a search format matched with the prompt word template according to a preset prompt word template to construct the target search result.

In some embodiments, the updating the target search result according to the third target document segmentation segment and the question text data comprises converting the third target document segmentation segment and the question into a search format matched with a preset prompt word template according to the preset prompt word template so as to update the target search result.

In a second aspect, the present application provides a question-answering processing system, the system comprising:

The first retrieval module is used for responding to the fact that the problem text data are detected to be input into the processing model, and determining target first types matched with the problem text data in a first vector database according to a first preset rule;

the second retrieval module is used for determining a target second category matched with the problem text data and a plurality of pending document segmentation fragments according to a second preset rule, the target first category and index information in the first vector database;

The third retrieval module is used for carrying out similarity calculation on the problem text data and the plurality of undetermined document segmentation fragments so as to select a first target document segmentation fragment;

And the answer generation module is used for constructing a target search result according to the first target document segmentation segment and the question text data, and inputting the target search result into a processing model to output an answer matched with the question text data.

In a third aspect, the present application provides a computer program which, when executed by a processor, performs the steps of:

In a fourth aspect, the present application provides an electronic device, including:

One or more processors;

And a memory associated with the one or more processors, the memory for storing program instructions that, when read and executed by the one or more processors, perform the operations of:

In a fifth aspect, the present application also provides a computer-readable storage medium having stored thereon a computer program that causes a computer to perform the operations of:

The beneficial effects achieved by the application are as follows:

The application provides a question-answer processing method, which comprises the steps of responding to the fact that question text data are detected to be input into a processing model, determining a target first category matched with the question text data in a first vector database according to a first preset rule, determining a target second category matched with the question text data and a plurality of undetermined document segmentation fragments according to a second preset rule, the target first category and index information in the first vector database, performing similarity calculation on the question text data and the undetermined document segmentation fragments to select a first target document segmentation fragment, constructing a target retrieval result according to the first target document segmentation fragment and the question text data, and inputting the target retrieval result into the processing model to output an answer matched with the question text data. By constructing the hierarchical vector database and subdividing the categories in the hierarchical database, more accurate related documents can be matched when the questions are searched, and the relevance between the text fragments obtained by searching and the questions is better, so that the accuracy and the reliability of model answers are improved.

Drawings

For a clearer description of the technical solutions of the embodiments of the present application, the drawings that are needed in the description of the embodiments will be briefly introduced below, it being obvious that the drawings in the description below are only some embodiments of the present application, and that other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art, wherein:

fig. 1 is a schematic flow chart of a question-answering processing method provided by an embodiment of the application;

FIG. 2 is a schematic diagram of a method for constructing a first vector database according to an embodiment of the present application;

FIG. 3 is a schematic diagram of a method for updating a target search result according to an embodiment of the present application;

FIG. 4 is a schematic diagram of a question-answering processing system according to an embodiment of the present application;

fig. 5 is a block diagram of an electronic device according to an embodiment of the present application.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application, and it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments of the present application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.

It should be understood that throughout this specification and the claims, unless the context clearly requires otherwise, the words "comprise", "comprising", and the like, are to be construed in an inclusive sense rather than an exclusive or exhaustive sense, that is, in the sense of "including but not limited to".

It should also be appreciated that the terms "first," "second," and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. Furthermore, in the description of the present application, unless otherwise indicated, the meaning of "a plurality" is two or more.

It should be noted that the terms "S1", "S2", and the like are used for the purpose of describing the steps only, and are not intended to be construed to be specific as to the order or sequence of steps, nor are they intended to limit the present application, which is merely used to facilitate the description of the method of the present application, and are not to be construed as indicating the sequence of steps. In addition, the technical solutions of the embodiments may be combined with each other, but it is necessary to base that the technical solutions can be realized by those skilled in the art, and when the technical solutions are contradictory or cannot be realized, the combination of the technical solutions should be considered to be absent and not within the scope of protection claimed in the present application.

As shown in a flow chart of a question-answering processing method in fig. 1, an embodiment of the present application provides a question-answering processing method, and specifically, the method disclosed in the embodiment of the present application is used to answer an input question, including the following contents:

S1, in response to detecting that the problem text data are input into a processing model, determining a target first category matched with the problem text data in a first vector database according to a first preset rule.

The processing model can be a trained large language model (LLM, large Language Model) which enables a machine to better understand and generate natural language text of a human being through a deep learning-based natural language processing technology. The application can realize dialogue with the processing model by using the large language model as the processing model, thereby obtaining the content and the answer which want to be known. It will be appreciated that the above-described processing model is not limited to large language models, but includes any other model that can be learned to achieve a matching answer to the question text data output.

The above problem text data in the application can be a problem or can be used for the text identification of the query; the question text data may be entered directly by the user or may be converted from other means, such as language input.

It can be appreciated that in order to enhance the retrieval effect, when the input of the question text data is detected, in some implementation scenarios, the question text data may be enhanced, and the question text data may be complemented, expanded, and enhanced, so as to further improve the comprehensiveness and usability of the answer finally generated by the processing model. Specifically, keywords of the problem text data can be extracted to obtain a plurality of undetermined keywords, keyword information is obtained after the undetermined keywords are filtered, and only a part of keywords which are ranked at the front in the problem text data are reserved as the keyword information; specifically, if keywords of the term type exist in the question text data, explanation of the term is required, wherein the term explanation content can be obtained through an existing term knowledge base. For keywords with English abbreviations in the question text data, the keywords need to be subjected to English abbreviation completion. If the term or English abbreviation is not involved in the question text data, the corresponding enhancement processing is directly carried out. And then carrying out problem enhancement processing on the problem text data according to the term interpretation content and/or the term full-name content, namely carrying out problem enhancement processing on the problem text data according to the term interpretation content, the term full-name content and the target supplementary keyword to obtain the enhanced problem text data. The target supplementary keywords are determined according to a preset knowledge keyword database and each keyword information in the question text data, wherein the preset knowledge keyword database is constructed according to sample supplementary keywords corresponding to sample keyword information, and the sample supplementary keywords at least comprise principle keywords, explanation keywords and sample keywords.

In a specific implementation scenario, determining the target first category matched with the question text data in the first vector database according to a first preset rule includes extracting a question feature of the question text data, and inputting the question feature into a classification model to output the target first category. Or obtaining keywords of the problem text data and pre-labeled domain information, and determining the target first category matched with the problem text data by matching the keywords of the problem with the keywords of the original document matched with each first category, matching the domain information of the problem with the domain information of the original document matched with each first category and calculating the similarity of the problem with the title of the original document matched with each first category. The method and the device can automatically generate the field information matched with the input problem text data by calling the annotation model by training the annotation model based on deep learning in advance, and do not expand the training process of the annotation model. When searching is carried out on input question text data, firstly, carrying out preliminary searching in a first vector database according to a first preset rule, carrying out preliminary searching according to the question features extracted from the question text data and basic features contained in an original document, reducing the range of fine searching in the later period, and accelerating the overall searching efficiency, thereby improving the answer speed of a processing model.

The classification model is used for determining a first class corresponding to an original document when the first vector database is constructed, and can be a model for classifying the document based on deep learning, and at the moment, problem characteristics of the problem text data are extracted, wherein the problem characteristics are characteristic vectors generated after vectorizing characters, words or sentences with substantial meanings in an input problem. By inputting the problem features into the trained classification model, a first classification corresponding to the problem features may be output. The training process of the classification model is not expanded any more.

When the first vector database is constructed, the first classification processing is performed on the original documents based on the classification model to generate a plurality of first categories, and it can be understood that the original documents under each category are identical or similar in domain information, and the keywords contained in the text and the title between the original documents have high repetition rate. The method comprises the steps of selecting an original document with high matching degree with question text data according to the matching of keywords of the question text data and keywords contained in the original document corresponding to each category, selecting the original document with high matching degree with the question text data according to the matching of the field information of the question text data and the field information contained in the original document, selecting the original document with high matching degree with the question text data according to the similarity of titles of the question text data and the original document, selecting the original document with high matching degree with the question text data according to the similarity ranking, and further determining the question text data according to the three matching methods. It should be noted that, the above three matching methods are not limited in order of execution, and may be executed in a certain order or simultaneously. In order to improve the retrieval efficiency, when the first classification processing is performed based on the classification model, a keyword set with high repeatability corresponding to each first category may be recorded, the title may be enhanced to obtain an enhanced title that may represent the first category (i.e. to implement integration of the same or similar title), and the domain information may be enhanced to obtain domain information that may represent the first category (i.e. to implement integration of the same or similar title), when the problem data text is mapped to the first category, keyword matching may be directly performed according to the keyword set corresponding to each first category and keywords included in the problem text data, matching may be performed according to the enhanced domain information corresponding to each first category and the enhanced domain information included in the problem text data, and similarity calculation may be performed according to the enhanced title corresponding to each first category and the problem text data, so as to directly determine the target first category with the highest matching degree with the problem text data.

S2, determining a target second category matched with the problem text data and a plurality of pending document segmentation fragments according to a second preset rule, the target first category and index information in the first vector database.

In order to facilitate hierarchical retrieval, the application also provides a method for recording a first mapping relation between a first category and a second category when the first vector database is constructed, recording a second mapping relation between the second category and the first document segmentation segment, and generating index information of the first vector database according to the first mapping relation and the second mapping relation. It will be appreciated that the index information may be stored in a designated memory space in the first vector database or may be stored separately and provided with a query interface to facilitate handling model calls. The application records the classification information and generates the index information on the basis of pre-establishing the first vector database, thereby facilitating the rapid positioning of the document range needing further retrieval after the first class is determined and accelerating the retrieval speed.

In a specific implementation scenario, the determining the target second category and the plurality of undetermined document segmentation fragments matched with the problem text data according to the second preset rule, the target first category and index information in the first vector database includes determining the plurality of undetermined second categories corresponding to the target first category according to the index information, calculating Euclidean distance between the average vector value of the first vector corresponding to the problem text data and the undetermined second category, determining the undetermined second category corresponding to the smallest Euclidean distance as the target second category, and determining the document segmentation fragments contained in the target second category as undetermined document segmentation fragments. It can be understood that the first vector database is obtained when determining the second class based on the center point determined by the clustering algorithm, wherein the method for calculating the euclidean distance is a conventional technical means in the art, and the disclosure is not repeated herein. After confirming the target first category matched with the question text data, the application calculates which second category of the question text data belongs to under the first category again, determines the target second category with the smallest Euclidean distance by a method for calculating the Euclidean distance, improves the accuracy of confirming the second category, improves the retrieval accuracy, and further reduces the calculation range to improve the retrieval efficiency. It can be appreciated that in some implementation scenarios with low requirements on search accuracy and high requirements on search comprehensiveness, the second category to be determined corresponding to the calculated euclidean distance smaller than the preset threshold may be determined as the target second category.

S3, similarity calculation is conducted on the problem text data and the plurality of pending document segmentation fragments to select a first target document segmentation fragment.

The method comprises the steps of obtaining text vectors matched with a to-be-determined document segmentation segment from a first vector database, calculating the similarity between the first vector and the text vectors, selecting a preset number of target text vectors according to the ranking of the calculated similarity, and determining the to-be-determined document segmentation segment corresponding to the target text vectors as the first target document segmentation segment. The similarity calculation of the first vector and the text vector can be determined by calculating cosine similarity between the two vectors, wherein the larger the cosine similarity is, the higher the similarity is, or by calculating Euclidean distance between the two vectors, the smaller the Euclidean distance is, the higher the similarity is, or by calculating Manhattan distance between the two vectors, the smaller the Manhattan distance is, the higher the similarity is, or by calculating Chebyshev distance, minkowski distance, etc. between the two vectors. The preset number is set by those skilled in the art according to the actual scene requirement and the search requirement, for example, set to 5 or 10, which is not limited in the present application.

S4, constructing a target search result according to the first target document segmentation segment and the question text data, and inputting the target search result into the processing model to output an answer matched with the question text data.

The method for constructing the target search result according to the first target document segmentation fragments and the question text data comprises the steps of converting the first target document segmentation fragments and the questions into a search format matched with the prompt word template according to a preset prompt word template so as to construct the target search result. Specifically, if the term template is a term template, a format such as < question > question (question >), and < content > target search result (content >), and the term template is used to tell where the model is the question, where the model is the search result, and then the model is allowed to answer question according to the content.

By constructing a hierarchical database and using a hierarchical search strategy when searching related documents for problems fed back by users, the search results which are more related to the problems are finally searched through continuous refined search and deep layer by layer, so that the usability and accuracy of the search results are improved.

In some implementation scenarios for enhancing the question text data, the step S4 is adaptively changed to construct a target search result according to the first target document segmentation segment and the enhanced question text data, and input the target search result to the processing model to output an answer matched with the question text data, and the specific content refers to the foregoing.

Fig. 2 is a schematic diagram of a method for constructing a first vector database according to the present application, where on the basis of the foregoing embodiment, the method for constructing a first vector database according to the present application includes the following steps:

The method comprises the steps of carrying out first classification processing on original documents according to basic characteristics of a plurality of original documents contained in an original document set to achieve classification of the original documents in the original document set into a plurality of first categories, wherein the original document set comprises one or more of private data or internet data of enterprises, obtaining document summaries of the original documents matched with each first category, wherein extraction of the document summaries can be generated by using a large language model or can be generated by training other learning models, carrying out second classification processing on the document summaries of the original documents in the first categories to achieve classification of the original documents in each first category into a plurality of second categories, segmenting the original documents in each second category to generate a plurality of first document segmentation fragments, and constructing a first vector database according to the first document segmentation fragments. After the first document segmentation segment is obtained, vector model can be utilized to vector the first document segmentation segment, and a first vector database is built based on the vectorized first document segmentation segment; the vectorization model can be integrated in the first vector database, the first document segmentation fragments are directly stored in the first vector database, and then vectorization of the first document segmentation fragments is automatically achieved by the first vector database. It can be appreciated that the conversion of the first document segmentation segment into a vector representation is achieved by inputting the first document segmentation segment into the vectorization model, and selecting an appropriate vectorization model, such as a word embedding model, in particular according to the task requirements and the data characteristics. Likewise, the basic feature acquisition disclosed by the application is realized through a vectorization model.

Specifically, the first classification processing is performed on the original documents according to the basic features of the original documents contained in the original document set to achieve classification of the original documents in the original document set into a plurality of first categories, wherein the first classification processing comprises the steps of obtaining basic information of the original documents contained in the original document set, including but not limited to keywords, titles and pre-labeled field information, extracting the basic features from the basic information, and inputting the basic features of the original documents contained in the original document set into a pre-trained classification model to output a plurality of first categories matched with the original document set. The method comprises the steps of obtaining a classification model, obtaining the information of the keywords, the titles and the fields of the documents, obtaining the information of the keywords, the fields and the fields of the keywords, the titles and the fields of the documents, obtaining the information of the keywords, the fields of the keywords, the titles and the fields of the documents, the fields of the keywords, the title and the field of the documents, the information, the title and the field of the information, the.

The classification model can be a model based on deep learning, and the classification model of the document is finally obtained to classify the document by extracting basic features in the original document and the pre-labeled field information to train the model. The basic feature is a feature vector vectorized according to words, sentences or segments having a substantial meaning with the original document. The specific classification model can be trained based on FastText models, textCNN models and TextRNN models, and can also be based on other models which can be used for text classification, and the specific selected models are not limited by the application. Notably, the classification model used in determining the first category matching the question text data as disclosed in the above embodiments is the same as the classification model used in constructing the first vector database. The domain information is information that a person skilled in the art marks an original document in advance, for example, a document in a hardware domain, and the domain classification can include principle design, layout and wiring, biso, bmc and the like, which are the domain information, and can also be realized by training a marking model as described above.

Specifically, the method comprises the steps of performing second classification processing on the document summaries of the original documents in the first categories to classify the original documents in each first category into a plurality of second categories, wherein the method comprises the steps of vectorizing the document summaries matched with each first category to generate summary vectors, and performing second classification processing on the summary vectors matched with each first category according to a clustering algorithm to classify the document summaries matched with each first category into a plurality of second categories. It can be determined that the second category to which the original document matching the document digest belongs can be determined according to the second category to which the document digest belongs. The vectorizing of the document abstract is as above, and may be implemented by using a vectorizing model, which is not described herein. The clustering algorithm is a k-means clustering algorithm, and in some implementation scenarios, other types of clustering algorithms, such as hierarchical clustering algorithm, partition clustering algorithm, and the like, are also used for performing adaptive processing. The k-means clustering algorithm is adopted to cluster a plurality of documents belonging to a first class, so that subdivision of small classes is realized, smaller subdivision classes can be searched when question searching is carried out, reliability and relevance of the searched documents are improved to a certain extent, the basis of a large model when answering questions is more accurate and reliable, and the obtained answers are more accurate and reliable.

It can be understood that the k-means clustering algorithm needs to preset the number of clustered categories, wherein the number of clustered categories in each first category is the number of second categories, specifically, the number of categories is determined by selecting and calculating whether any two abstract vectors are orthogonal from abstract vectors matched in the first category, if the two abstract vectors are orthogonal, defining an intersection point of the two abstract vectors as an initialization center point, repeatedly executing the steps of acquiring the initialization center point until the first category matched abstract vectors are traversed, and determining the number of second categories under the first category according to the number of the initialization center points. That is, to avoid excessive human intervention, we map the document abstract to the vector space, traverse all feature vectors, calculate whether every two feature vectors are orthogonal, if so, set it as an initialization center point, repeat the work until all the initialization center points are selected and K are recorded, and finally divide the document abstract into K classes, that is, the number of the second classes of the first class subordinate obtained by performing the second classification processing on the original documents of each first class subordinate is determined.

On the basis of the above embodiment, as shown in a schematic diagram of a target search result updating method disclosed in fig. 3, the present application also discloses a target search result updating method, which specifically includes the following contents:

And X1, responding to the fact that the problem text data are detected to be input into the processing model, and acquiring a first vector matched with the problem text data.

Specifically, the first vector is obtained by inputting the text data of the problem into a vectorization model for conversion, and specifically, a proper vectorization model, such as a word embedding model, is selected according to task requirements and data characteristics. Similarly, the candidate problem is subsequently vectorized to generate a second vector by converting the candidate problem into a vector model.

And X2, searching a second vector matched with the first vector in a second vector database.

The construction method of the second vector database comprises the steps of obtaining all original documents contained in an original document set, segmenting the original documents to generate a plurality of second document segmentation fragments, generating alternative problems matched with the second document segmentation fragments, vectorizing the alternative problems to generate a second vector, and constructing the second vector database according to the second vector. By converting the document data into possible problems in advance and constructing a second vector database based on the problems, the retrieval is realized from the problem level, the retrieval level is enriched, and the retrieval accuracy is improved. In a specific implementation scenario, the candidate problem of the second document segmentation segment matching can be generated by adopting a large language model, and the candidate problem can also be realized by training a model based on deep learning in advance, so that the application is not limited herein.

And X3, determining a second target document segmentation segment matched with the second vector according to the corresponding relation between the document segmentation segment and the second vector.

It will be appreciated that since the second vector is generated by vectorizing the document segmentation fragments, there is a one-to-one correspondence between the document segmentation fragments and the second vector. Wherein the determined number of the second target document segmentation fragments corresponds to the above-mentioned preset number.

And X4, screening the first target document segmentation segment and the second target document segmentation segment to generate a third target document segmentation segment.

The method for confirming the first target document segmentation segment is the same as steps S1-S3 disclosed in the foregoing embodiment, and the present application is not described herein again.

Specifically, the step of screening the first target document segmentation segment and the second target document segmentation segment to generate a third target document segmentation segment may be performed by removing from high to low according to the obtained similarity value, and obtaining document segmentation segments meeting the number requirements after removing the repeated condition, wherein the step of ranking the first target document segmentation segment and the second target document segmentation segment only includes one document segmentation segment repeated in the first target document segment and the second target document segmentation segment according to the calculated similarity value between the text vector corresponding to the first target document segmentation segment and the first vector and the calculated similarity value between the second vector corresponding to the second target document segmentation segment and the first vector. Since the similarity criteria are the same, the similarity calculation mode adopted in the method is consistent, and therefore, the document segmentation fragments meeting the preset number can be selected as the third target document segmentation fragments according to the ranking level, wherein the preset number is set in the above description, and the description is omitted.

In order to improve the accuracy and pertinence of a finally constructed target retrieval result, the application further provides adding a retrieval rearrangement model, sequencing the acquired plurality of first target document segmentation fragments and second target document segmentation fragments based on a preset retrieval rearrangement model and problem text data, taking the document segmentation fragments corresponding to N results which are sequenced in front as third target document segmentation fragments according to sequencing results, wherein N is a quantity value determined by a person skilled in the art according to an actual scene and retrieval requirements. The search rearrangement model used may be selected according to the situation, for example, bge-reranker-large search rearrangement model may be selected in the present application, which is not limited herein.

And X5, updating the target retrieval result according to the third target document segmentation fragment and the problem text data.

Specifically, according to a preset prompting word template, the third target document segmentation segment and the problem are converted into a retrieval format matched with the prompting word template so as to construct a new target retrieval result. The format of < question > question question >, < content > target search result question > is used to tell the model where the question is and where the search result is, and then the model is allowed to answer question according to the content.

In the embodiment, when the hierarchical database is constructed and related documents are searched for the user questions, the hierarchical searching strategy is used for continuously refining the search, the related questions generated based on the documents are used for constructing the second vector database, then the question-to-question vector matching is adopted, the searching accuracy is improved, and finally the two search results are combined, so that the accuracy and usability of answering the professional field questions by the model are improved to a certain extent.

On the basis of the above embodiment, as shown in the system architecture diagram of fig. 4, an embodiment of the present application further provides a question-answering processing system, including:

A first retrieval module 410 for determining, in response to detecting that there is a question text data input to the processing model, a target first category matching the question text data according to a first preset rule in a first vector database;

A second search module 420, configured to determine a target second category and a plurality of pending document segmentation fragments that match the question text data according to a second preset rule, a target first category, and index information in the first vector database;

A third retrieving module 430, configured to perform similarity calculation on the question text data and the plurality of pending document segmentation fragments to select a first target document segmentation fragment;

and the answer generation module 440 is configured to construct a target search result according to the first target document segmentation segment and the question text data, and input the target search result to a processing model to output an answer matched with the question text data.

In some implementations, the system further includes a problem retrieval module (not illustrated in the figure) configured to obtain a first vector matching the problem text data in response to detecting that the problem text data is input to the processing model, search a second vector database for a target vector matching the first vector, determine a second target document segmentation segment matching the second vector according to a correspondence between the document segmentation segment and the second vector, wherein the third retrieval module 430 is further configured to screen the first target document segmentation segment and the second target document segmentation segment to generate a third target document segmentation segment, and the answer generation module 440 is further configured to update a target retrieval result according to the third target document segmentation segment and the problem text data.

In some implementations, the system includes a data preparation module (not shown) configured to construct a first vector database, perform a first classification process on the original documents according to basic features of a plurality of original documents included in an original document set to classify the original documents in the original document set into a plurality of first categories, obtain document summaries of the original documents matching each of the first categories, perform a second classification process on the document summaries of the original documents in the first categories to classify the original documents in each of the first categories into a plurality of second categories, segment the original documents in each of the second categories to generate a plurality of first document segmentation segments, and construct the first vector database according to the first document segmentation segments.

In some implementation scenarios, the data preparation module is further used for constructing a construction method of a second vector database, wherein the construction method comprises the steps of obtaining all original documents contained in an original document set, segmenting the original documents to generate a plurality of second document segmentation fragments, generating alternative questions matched with the second document segmentation fragments, vectorizing the alternative questions to generate the second vector, and constructing the second vector database according to the second vector.

In some implementations, the data preparation module is further configured to record a first mapping relationship between the first category and the second category, record a second mapping relationship between the second category and the first document segmentation segment, and generate index information of the first vector database according to the first mapping relationship and the second mapping relationship.

In some implementations, the data preparation module is further configured to obtain basic information of a plurality of original documents included in the set of original documents and extract basic features from the basic information, and input the basic features of the plurality of original documents included in the set of original documents into a classification model to output a plurality of first categories matching the set of original documents.

In some implementations, the data preparation module is further configured to vectorize the document summaries of each of the first category matches to generate summary vectors, and perform a second classification process on the summary vectors of each of the first category matches according to a clustering algorithm to classify the document summaries of each of the first category matches into a plurality of second categories.

In some implementation scenarios, the data preparation module is further configured to select and calculate whether any two abstract vectors are orthogonal from the abstract vectors matched in the first category, define an intersection point of the two abstract vectors as an initialization center point if the two abstract vectors are orthogonal, repeatedly perform the step of acquiring the initialization center point until the abstract vectors matched in the first category are traversed, and determine the number of second categories under the first category according to the number of the initialization center points.

In some implementations, the first retrieval module 410 is configured to extract the problem feature of the problem text data and input the problem feature into the classification model to output a target first category, or obtain a keyword of the problem text data and pre-labeled domain information, match the domain information of the problem text data with the domain information of the original document matched with each first category by matching the keyword of the problem text data with the keyword of the original document matched with each first category, and calculate the similarity of the problem text data with the title of the original document matched with each first category, and determine the target first category matched with the problem text data.

In some implementations, the second search module 420 is configured to confirm a plurality of second classes to be determined corresponding to the target first class according to the index information, calculate a euclidean distance between a first vector corresponding to the text data of the question and an average vector value of the second classes to be determined, determine the second class to be determined corresponding to the smallest euclidean distance as the target second class, and determine a first document segmentation segment included in the target second class as the second document segmentation segment to be determined.

In some implementations, the third retrieving module 430 is configured to obtain a text vector matching the pending document segmentation segment from the first vector database, calculate a similarity between the first vector and the text vector, select a preset number of target text vectors according to the ranking of the calculated similarity, and determine the pending document segmentation segment corresponding to the target text vectors as a first target document segmentation segment.

In some implementations, the answer generation module 440 is configured to convert the first target document segmentation segment and the question into a search format matched with the prompt word template according to a preset prompt word template, so as to construct the target search result.

In some implementations, the answer generation module 440 is configured to convert the third target document segmentation segment and the question into a search format matching the prompt word template according to a preset prompt word template to update the target search result.

On the basis of the above embodiments, the embodiments of the present application further provide a computer program, which when executed by a processor, implements the steps of the method of:

Corresponding to all the embodiments described above, the embodiment of the application provides an electronic device comprising one or more processors, and a memory associated with the one or more processors for storing program instructions which, when read for execution by the one or more processors, perform the following operations:

Fig. 5 illustrates an architecture of an electronic device, which may include a processor 510, a video display adapter 511, a disk drive 512, an input/output interface 513, a network interface 514, and a memory 520, among others. The processor 510, the video display adapter 511, the disk drive 512, the input/output interface 513, the network interface 514, and the memory 520 may be communicatively connected by a bus 530.

The processor 510 may be implemented by a general-purpose CPU (central processing unit), a microprocessor, an application-specific circuit (ApplicationSpecificIntegratedCircuit, ASIC), or one or more circuits, etc., for executing related programs to implement the technical solution provided by the present application.

The memory 520 may be implemented in the form of ROM (read only memory), RAM (random access memory), a static storage device, a dynamic storage device, or the like. The memory 520 may store an operating system 521 for controlling the execution of the electronic device 500, and a Basic Input Output System (BIOS) 522 for controlling the low-level operation of the electronic device 500. In addition, a web browser 523, a data storage management system 524, an icon font processing system 525, and the like may also be stored. The icon font processing system 525 may be an application program that implements the operations of the foregoing steps in the embodiment of the present application. In general, when the technical solution provided by the present application is implemented by software or firmware, relevant program codes are stored in the memory 520 and invoked by the processor 510 to be executed.

The input/output interface 513 is used for connecting with an input/output module to realize information input and output. The input/output module may be configured as a component in a device (not shown) or may be external to the device to provide corresponding functionality. Wherein the input devices may include a keyboard, mouse, touch screen, microphone, various types of sensors, etc., and the output devices may include a display, speaker, vibrator, indicator lights, etc.

The network interface 514 is used to connect communication modules (not shown) to enable communication interactions of the device with other devices. The communication module may implement communication through a wired manner (such as USB, network cable, etc.), or may implement communication through a wireless manner (such as mobile network, WIFI, bluetooth, etc.).

Bus 530 includes a path to transfer information between components of the device (e.g., processor 510, video display adapter 511, disk drive 512, input/output interface 513, network interface 514, and memory 520).

In addition, the electronic device 500 may also obtain information of specific acquisition conditions from the virtual resource object acquisition condition information database, for performing condition judgment, and so on.

It should be noted that although the above devices only show the processor 510, the video display adapter 511, the disk drive 512, the input/output interface 513, the network interface 514, the memory 520, the bus 530, etc., in the specific implementation, the device may include other components necessary to achieve normal execution. Furthermore, it will be appreciated by those skilled in the art that the apparatus may include only the components necessary to implement the present application, and not all of the components shown in the drawings.

From the above description of embodiments, it will be apparent to those skilled in the art that the present application may be implemented in software plus a necessary general hardware platform. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a storage medium, such as a ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer device (which may be a personal computer, a cloud server, or a network device, etc.) to execute the method of the embodiments or some parts of the embodiments of the present application.

Corresponding to all the embodiments described above, the embodiments of the present application also provide a computer-readable storage medium storing a computer program causing a computer to execute the operations of:

In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for a system or system embodiment, since it is substantially similar to a method embodiment, the description is relatively simple, with reference to the description of the method embodiment being made in part. The systems and system embodiments described above are merely illustrative, wherein elements illustrated as separate elements may or may not be physically separate, and elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.

The foregoing is only illustrative of the present application and is not to be construed as limiting thereof, but rather as various modifications, equivalent arrangements, improvements, etc., within the spirit and principles of the present application.

Claims

1. A question-answering processing method, characterized in that the method comprises:

2. The method of claim 1, further comprising updating the target search result, the updating method comprising:

and updating the target retrieval result according to the third target document segmentation segment and the problem text data.

3. The method of claim 1, further comprising the steps of constructing a first vector database:

Acquiring a document abstract of each first-class matched original document;

4. The method of claim 2, further comprising a second vector database construction method of:

5. A method according to claim 3, wherein after constructing a first vector database from the first document segmentation fragments, the method comprises:

6. The method of claim 3, wherein said performing a first classification process on said original documents based on basic features of a plurality of original documents contained in a set of original documents to achieve classification of the original documents in the set of original documents into a plurality of first categories comprises:

7. A method according to claim 3, wherein said performing a second classification of the document summaries of the original documents in the first categories to classify the original documents in each of the first categories into a plurality of second categories comprises:

8. The method of claim 7, wherein when classifying each of the first class-matched summary vectors into a plurality of second classes according to a clustering algorithm, the determining method of the number of second classes under the first class comprises:

9. The method according to any of claims 6-8, wherein said determining in the first vector database a target first category matching the question text data according to a first preset rule comprises:

Acquiring keywords of the problem text data and pre-labeled field information;

10. The method of claim 9, wherein determining a target second category and a plurality of pending document segmentation fragments matching the question text data based on a second preset rule, a target first category, and index information in the first vector database, comprises:

11. The method of claim 10, wherein performing similarity calculation on the question text data and the plurality of pending document snippets to select a first target document snippet comprises:

calculating the similarity between the first vector and the text vector;

12. The method of claim 4, wherein constructing a target search result from the first target document segmentation segment and the question text data comprises converting the first target document segmentation segment and the question into a search format matching the hint word template according to a preset hint word template to construct the target search result.

13. The method of claim 12, wherein updating the target search result based on the third target document segmentation segment and the question text data comprises converting the third target document segmentation segment and the question into a search format matching the hint word template based on a preset hint word template to update the target search result.

14. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1-13.

15. An electronic device, the electronic device comprising:

One or more processors;

and a memory associated with the one or more processors, the memory for storing program instructions that, when read for execution by the one or more processors, perform the method of any of claims 1-13.

16. A computer readable storage medium, characterized in that it stores a computer program, which causes a computer to perform the method of any one of claims 1-13.