CN118035807A - Evaluation method and device of large language model, electronic equipment and storage medium - Google Patents

Abstract

The present invention provides a method, device, electronic device and storage medium for evaluating a large language model, and relates to the field of artificial intelligence technology. The method includes: obtaining interaction information to be tested corresponding to the large language model to be tested, and a first manual score of the user scoring the large language model to be tested based on a manual evaluation template; the manual evaluation template is a scoring template determined based on at least two subjective indicators; the interaction information to be tested is input into at least two evaluation models, and a first objective score of each evaluation model scoring the large language model to be tested is output; the loss function of each evaluation model is determined based on a weighted average of at least two objective indicators; based on the first manual score and each first objective score, a global score corresponding to the large language model to be tested is determined. The present invention can improve the objectivity and comprehensiveness of the evaluation of the large language model to be tested, and thus improve the reliability of the evaluation result of the large language model to be tested.

Description

Large language model evaluation method, device, electronic device and storage medium

Technical Field

The present invention relates to the field of artificial intelligence technology, and in particular to a large language model evaluation method, device, electronic device and storage medium.

Background technique

With the continuous development of deep learning and natural language processing technology, a large number of large language models (LLM) have emerged. Evaluating large language models can further evaluate their performance on specific tasks, identify their shortcomings, and facilitate targeted improvements to existing large language models, thereby improving their performance.

In the prior art, a large language model is generally used to interact with users, and the users rate the performance of the large language model in the interaction process for each rating item, and the large language model is evaluated through manual scoring. However, manual scoring is relatively subjective, resulting in low objectivity in the evaluation of the large language model, which in turn leads to low reliability in the evaluation results of the large language model.

Summary of the invention

The present invention provides a large language model evaluation method, device, electronic device and storage medium, which are used to solve the defect of low reliability of evaluation results of large language models in the prior art.

The present invention provides a large language model evaluation method, comprising:

Acquire the interaction information to be tested corresponding to the large language model to be tested, and a first manual score of the large language model to be tested by a user based on a manual evaluation template; the manual evaluation template is a scoring template determined based on at least two subjective indicators;

Inputting the interaction information to be tested into at least two evaluation models, and outputting a first objective score of each evaluation model for scoring the large language model to be tested; the loss function of each evaluation model is determined based on a weighted average of at least two objective indicators;

A global score corresponding to the large language model to be tested is determined based on the first manual score and each of the first objective scores.

According to the large language model evaluation method provided by the present invention, the step of inputting the interaction information to be tested into at least two evaluation models and outputting a first objective score of the large language model to be tested by each evaluation model includes:

For each of the evaluation models, the interaction information to be tested is input into the evaluation model to determine prediction information corresponding to the interaction information to be tested; the prediction information is used to characterize the correctness of the interaction information to be tested;

Based on the prediction information, determining an indicator score corresponding to each of the objective indicators;

Based on the scores of the indicators and the score weights corresponding to the scores of the indicators, a first objective score of the evaluation model for scoring the large language model to be tested is determined.

According to the large language model evaluation method provided by the present invention, determining the global score corresponding to the large language model to be tested based on the first manual score and each of the first objective scores includes:

Obtaining a score weight corresponding to each of the evaluation models, wherein the score weight is determined based on a ratio of an amount of sub-training data corresponding to each of the evaluation models to an amount of total training data corresponding to all the evaluation models;

Determining a second objective score corresponding to the large language model to be tested based on each of the score weights and the first objective score of the evaluation model corresponding to each of the score weights;

A global score corresponding to the large language model to be tested is determined based on a weighted sum of the first manual score and the second objective score.

According to the large language model evaluation method provided by the present invention, determining the global score corresponding to the large language model to be tested based on the weighted sum of the first manual score and the second objective score includes:

Performing semantic analysis on the interaction information to be tested to determine target domain information corresponding to the interaction information to be tested;

Based on the target domain information, determining a target manual scoring weight corresponding to the first manual scoring and a target objective scoring weight corresponding to the second objective scoring from a preset mapping relationship; the preset mapping relationship includes a mapping relationship between domain information, manual scoring weights, and objective scoring weights;

A global score corresponding to the large language model to be tested is determined based on the first manual score, the target manual score weight, the second objective score, and the target objective score weight.

According to the evaluation method for a large language model provided by the present invention, the objective indicators corresponding to the maximum scoring weights in the loss functions of the evaluation models are different.

The large language model evaluation method provided by the present invention obtains a first manual score of the large language model to be tested by the user based on the manual evaluation template, comprising:

Obtaining a second subjective score corresponding to each of the subjective indicators by the user;

determining a sum of all second subjective ratings;

Based on the ratio of the sum of all the second subjective scores to the number of all the subjective indicators, a first manual score of the user for scoring the large language model to be tested based on the manual evaluation template is determined.

The method for evaluating a large language model according to the present invention further includes:

For each of the subjective indicators, determine the number of scoring points corresponding to the subjective indicator;

Based on the number of the scoring values, determining the degree of the subjective indicator corresponding to each of the scoring values;

Based on all the scoring scores and the degree of the subjective indicator corresponding to each of the scoring scores, construct a sub-evaluation template corresponding to the subjective indicator;

Constructing the manual evaluation template based on the sub-evaluation templates corresponding to all subjective indicators;

The manual evaluation template and the interaction information to be tested are sent to a terminal corresponding to the user.

The present invention also provides a large language model evaluation device, comprising:

An acquisition module, configured to acquire interaction information to be tested corresponding to the large language model to be tested, and a first manual score of the large language model to be tested by a user based on a manual evaluation template; the manual evaluation template is a scoring template determined based on at least two subjective indicators;

An output module, used for inputting the interaction information to be tested into at least two evaluation models, and outputting a first objective score of each evaluation model for scoring the large language model to be tested; the loss function of each evaluation model is determined based on a weighted average of at least two objective indicators;

A determination module is used to determine a global score corresponding to the large language model to be tested based on the first manual score and each of the first objective scores.

The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the evaluation method for the large language model described above is implemented.

The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the large language model evaluation method described in any one of the above is implemented.

The large language model evaluation method, device, electronic device and storage medium provided by the present invention, after obtaining the interaction information to be tested and the first manual score of the large language model to be tested by the user, predict the interaction information to be tested by each evaluation model, output the first objective score of the large language model to be tested by each evaluation model, after combining the first manual score and each first objective score, obtain the global score of the large language model to be tested, after combining the subjective index and the objective index, can improve the objectivity and comprehensiveness of the evaluation of the large language model to be tested, thereby improving the reliability of the evaluation result of the large language model to be tested.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

FIG1 is a schematic diagram of a flow chart of a method for evaluating a large language model provided by an embodiment of the present invention;

FIG2 is a schematic diagram of the structure of a large language model evaluation device provided by an embodiment of the present invention;

FIG. 3 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention.

Detailed ways

In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

In view of the problem that the evaluation results of large language models in the prior art are of low reliability, an embodiment of the present invention provides an evaluation method for a large language model. FIG1 is a flow chart of the evaluation method for a large language model provided by an embodiment of the present invention. As shown in FIG1 , the method includes:

Step 110: Obtain the interaction information to be tested corresponding to the large language model to be tested, and a first manual score of the large language model to be tested by a user based on a manual score template; the manual score template is a score template determined based on at least two subjective indicators.

Optionally, the interaction information to be tested may be historical interaction texts of multiple rounds of interaction between the large language model to be tested and the user in the same field or in different fields, and the historical interaction texts include historical question texts input by the user and historical reply texts generated by the large language model based on the question texts. It should be noted that the interaction information to be tested is obtained after authorization by the user.

Optionally, the field may include the medical field, academic field, scientific field, artistic field, literary field, and financial field, etc., which is not limited in the embodiment of the present invention.

Optionally, the user who scores the large language model to be tested based on the manual evaluation template may be a real user who interacts with the large language model to be tested, or may be an evaluation expert, which is not limited in this embodiment of the present invention.

Optionally, at least two subjective indicators in the manual evaluation template can be understood as scoring items for manual scoring, and the subjective indicators can include: basic general capabilities, vertical field capabilities, distribution hit rate, relevance, irrelevance, correctness, interactivity and model performance, etc., among which: basic general capabilities can be understood as the relevance and sentence fluency of the reply text of the large language model to be tested and the question text; the vertical field capabilities can be understood as the relevance of the reply text of the large language model to be tested and the question text and the keywords involved in the relevant fields; the distribution hit rate can be understood as the number of keywords hit in the reply text; the relevance can be understood as the semantic relevance of the reply text and the number of keywords that meet the requirements; the irrelevance is the antonym of the relevance; the correctness is the correctness of the reply text judged by the user; the interactivity can be understood as the relevance of the reply text generated after a preset number of follow-up questions; the model performance can be understood as the ratio of the actual performance of the model to the expected performance of the model, and the model performance can include model response speed, interface return speed and QPS (Queries Per Second) indicators, etc.

In addition, the subjective indicator may also include the extent to which the reply text considers the emotions of the questioning user in the question text or the extent to which the reply text considers the identity data of the questioning user. The identity data may include the age, gender, education, social status and other data of the questioning user, and the embodiments of the present invention are not limited to this.

Furthermore, the method further comprises:

For each of the subjective indicators, determine the number of scoring points corresponding to the subjective indicator;

Based on the number of the scoring values, determining the degree of the subjective indicator corresponding to each of the scoring values;

Based on all the scoring scores and the degree of the subjective indicator corresponding to each of the scoring scores, construct a sub-evaluation template corresponding to the subjective indicator;

Constructing the manual evaluation template based on the sub-evaluation templates corresponding to all subjective indicators;

The manual evaluation template and the interaction information to be tested are sent to a terminal corresponding to the user.

For example, after determining the above subjective indicators, the scoring range can be determined as 0-5 points, and then the scoring range can be divided into 6 scoring values with 1 point as an interval, and the 6 scoring values are 0 points, 1 point, 2 points, 3 points, 4 points and 5 points. The degree of satisfying the subjective indicator from 0 points to 5 points in the 6 scoring values increases in sequence, that is, 0 points indicates the lowest degree of satisfying the subjective indicator, and 5 points indicates the highest degree of satisfying the subjective indicator. After determining the scoring values, for each subjective indicator, the degree of satisfying the subjective indicator is divided according to the number of scoring values, and the sub-evaluation template of the subjective indicator is constructed by combining the scoring values and the degree of satisfying the subjective indicator corresponding to each scoring value. Taking the subjective indicator as the basic general ability as an example, 0 points means that the reply text of the large language model to be tested is completely unrelated to the question text; 1 point means that the reply text of the large language model to be tested is unrelated to the question text, but the sentence of the reply text is slightly fluent; 2 points means that the reply text of the large language model to be tested is unrelated to the question text, but the sentence of the reply text is fluent; 3 points means that the reply text of the large language model to be tested is slightly related to the question text, and the sentence of the reply text is fluent; 4 points means that the reply text of the large language model to be tested is related to the question text, and the sentence of the reply text is fluent; 5 points means that the reply text of the large language model to be tested is completely related to the question text, and the sentence of the reply text is fluent and logical. After synthesizing all subjective indicators and the sub-evaluation templates corresponding to each subjective indicator, the manual evaluation template can be obtained. The manual evaluation template is shown in Table 1. In Table 1, except for the key fields corresponding to the first row, each row represents a subjective indicator and the sub-evaluation template corresponding to the subjective indicator.

Table 1

Further, obtaining a first manual score of the user for the large language model to be tested based on the manual evaluation template includes:

Obtaining a second subjective score corresponding to each of the subjective indicators by the user;

determining a sum of all second subjective ratings;

Based on the ratio of the sum of all the second subjective scores to the number of all the subjective indicators, a first manual score of the user for scoring the large language model to be tested based on the manual evaluation template is determined.

Specifically, different scoring values in each subjective indicator correspond to different second subjective scores, for example, 0 points correspond to 0 points, 1 point corresponds to 20 points, 2 points correspond to 40 points, 3 points correspond to 60 points, 4 points correspond to 80 points, and 5 points correspond to 100 points, that is, after determining the scoring value selected by the user, the scoring value selected by the user can be multiplied by 20 to obtain the second subjective score corresponding to the subjective indicator. After calculating the second subjective score corresponding to each subjective indicator, the sum of all second subjective scores can be calculated, and then the ratio of the sum to the number of all subjective indicators is calculated to obtain the first manual score in the range of 0-100 points.

Step 120: input the interaction information to be tested into at least two evaluation models, and output a first objective score of each evaluation model for scoring the large language model to be tested; the loss function of each evaluation model is determined based on a weighted average of at least two objective indicators.

Specifically, since only the first manual score will result in low reliability of the evaluation result of the large language model to be tested, in an embodiment of the present invention, after obtaining the interaction information to be tested, the interaction information to be tested is input into multiple evaluation models respectively, and each evaluation model classifies the interaction information to be tested to obtain prediction information of the interaction information to be tested, and determines the first objective score based on the prediction information.

Optionally, the objective indicators may include accuracy, precision, recall, and F1 score, among which: accuracy is used to measure the proportion of samples predicted by the model as positive samples that are truly positive samples. Precision is used to measure the proportion of samples predicted by the model as positive samples that are truly positive samples. Recall is used to measure the proportion of samples predicted by the model as positive samples that are truly positive samples. The F1 score is the harmonic mean of the precision and recall rates. The calculation formula corresponding to the F1 score is shown in formula (1):

Among them, F1_score represents the F1 score.

It should be noted that the objective indicators corresponding to the maximum score weights in the loss functions of the evaluation models are different. That is, the objective indicators evaluated by multiple evaluation models are the same, but each evaluation model focuses on different objective indicators. For example, taking the objective indicators including accuracy, precision, recall and F1 score, and the number of evaluation models is 4,

1) Evaluation model 1 focuses on accuracy. The loss function of evaluation model 1 is shown in formula (2):

Loss1＝ _ωa1 *Accuracy+ _ωb1 *Precision+ _ωc1 *Recall+ _ωd1 *F1_score

Among them, Loss1 represents the loss function of evaluation model 1, ω _a1 represents the scoring weight corresponding to the accuracy in evaluation model 1, ω _b1 represents the scoring weight corresponding to the precision in evaluation model 1, ω _c1 represents the scoring weight corresponding to the recall in evaluation model 1, ω _d1 represents the scoring weight corresponding to the F1 score in evaluation model 1, ω _a1 +ω _b1 +ω _c1 +ω _d1 = 1, and ω _a1 is greater than any of the scoring weights among ω _b1 , ω _c1 and ω _d1 .

2) Evaluation model 2 focuses on accuracy. The loss function of evaluation model 2 is shown in formula (3):

Loss2＝ _ωa2 *Accuracy+ _ωb2 *Precision+ _ωc2 *Recall+ _ωd2 *F1_score

Among them, Loss2 represents the loss function of evaluation model 2, ω _a2 represents the scoring weight corresponding to the accuracy in evaluation model 2, ω _b2 represents the scoring weight corresponding to the precision in evaluation model 2, ω _c2 represents the scoring weight corresponding to the recall in evaluation model 2, ω _d2 represents the scoring weight corresponding to the F1 score in evaluation model 2, ω _a2 +ω _b2 +ω _c2 +ω _d2 = 1, and ω _b2 is greater than any of the scoring weights among ω _a2 , ω _c2 and ω _d2 .

3) Evaluation model 3 focuses on recall rate. The loss function of evaluation model 3 is shown in formula (4):

Loss3＝ _ωa3 *Accuracy+ _ωb3 *Precision+ _ωc3 *Recall+ _ωd3 *F1_score

Among them, Loss3 represents the loss function of evaluation model 3, ω _a3 represents the scoring weight corresponding to the accuracy in evaluation model 3, ω _b3 represents the scoring weight corresponding to the precision in evaluation model 3, ω _c3 represents the scoring weight corresponding to the recall in evaluation model 3, ω _d3 represents the scoring weight corresponding to the F1 score in evaluation model 3, ω _a3 +ω _b3 +ω _c3 +ω _d3 = 1, and ω _c3 is greater than any one of the scoring weights among ω _a3 , ω _b3 and ω _d3 .

4) Evaluation model 4 focuses on the recall rate. The loss function of evaluation model 4 is shown in formula (5), which is:

Loss4＝ _ωa4 *Accuracy+ _ωb4 *Precision+ _ωc4 *Recall+ _ωd4 *F1_score

Among them, Loss4 represents the loss function of evaluation model 4, ω _a4 represents the scoring weight corresponding to the accuracy in evaluation model 4, ω _b4 represents the scoring weight corresponding to the precision in evaluation model 4, ω _c4 represents the scoring weight corresponding to the recall in evaluation model 4, ω _d4 represents the scoring weight corresponding to the F1 score in evaluation model 4, ω _a4 +ω _b4 +ω _c4 +ω _d4 = 1, and ω _d4 is greater than any one of the scoring weights among ω _a4 , ω _b4 and ω _c4 .

Optionally, when training each evaluation model, historical conversation data from different fields and labels corresponding to the historical conversation data can be obtained, and supervised training can be performed on each evaluation model. After obtaining the predicted categories output by each evaluation model, the parameters of the corresponding evaluation models are updated through the above-mentioned different loss functions, so that the evaluation performance of each evaluation model can be different. The label can be understood as the correctness corresponding to the historical conversation data. The label includes positive and negative categories. The positive category can indicate that the reply text of the large language model to be tested is related to the corresponding question text and is correct, and the negative category can indicate that the reply text of the large language model to be tested is irrelevant to the corresponding question text and is wrong.

Optionally, the amount of sub-training data of the training data corresponding to each evaluation model may be the same or different, and the embodiment of the present invention does not impose any limitation on this.

Furthermore, the step of inputting the interaction information to be tested into at least two evaluation models and outputting a first objective score of the large language model to be tested by each evaluation model includes:

For each of the evaluation models, the interaction information to be tested is input into the evaluation model to determine prediction information corresponding to the interaction information to be tested; the prediction information is used to characterize the correctness of the interaction information to be tested;

Based on the prediction information, determining an indicator score corresponding to each of the objective indicators;

Based on the scores of the indicators and the score weights corresponding to the scores of the indicators, a first objective score of the evaluation model for scoring the large language model to be tested is determined.

Specifically, after each evaluation model is trained, the interaction information to be tested is input into each evaluation model, and the prediction information corresponding to each evaluation model can be obtained. The prediction information can be understood as the prediction category of whether the interaction information to be tested is correct or not. After obtaining the prediction information, the indicator score corresponding to each objective indicator of the interaction information to be tested can be calculated, and the first objective score corresponding to each evaluation model is calculated according to the indicator score and the corresponding score weight. For example, the first objective score determined according to the prediction information output by the evaluation model 1 can be: Score _1-1 =ω _a1 *Accuracy ₁ +ω _b1 *Precision ₁ +ω _c1 *Recall ₁ +ω _d1 *F1_score ₁ , wherein Score _1-1 represents the first objective score corresponding to the evaluation model 1, Accuracy ₁ represents the accuracy rate corresponding to the evaluation model 1, Precision ₁ represents the precision rate corresponding to the evaluation model 1, Recall ₁ represents the recall rate corresponding to the evaluation model 1, and F1_score ₁ represents the F1 score corresponding to the evaluation model 1.

Step 130: Determine a global score corresponding to the large language model to be tested based on the first manual score and each of the first objective scores.

Specifically, after determining the first manual score and the first objective scores corresponding to each evaluation model, the global score corresponding to the large language model to be tested may be calculated according to a weighted sum of the first manual score and each first objective score.

Furthermore, determining a global score corresponding to the large language model to be tested based on the first manual score and each of the first objective scores includes:

Obtaining a score weight corresponding to each of the evaluation models, wherein the score weight is determined based on a ratio of an amount of sub-training data corresponding to each of the evaluation models to an amount of total training data corresponding to all the evaluation models;

Determining a second objective score corresponding to the large language model to be tested based on each of the score weights and the first objective score of the evaluation model corresponding to each of the score weights;

A global score corresponding to the large language model to be tested is determined based on a weighted sum of the first manual score and the second objective score.

Specifically, after determining each first objective score, the score weight corresponding to each first objective score can be determined according to the ratio of the sub-training data volume of the training data corresponding to each evaluation model to the total training data volume of all training data. For example, if all training data are 10,000, the sub-training data volume of the training data corresponding to evaluation model 1 is 3,000, then the score weight corresponding to evaluation model 1 is 0.3, the sub-training data volume of the training data corresponding to evaluation model 2 is 2,000, then the score weight corresponding to evaluation model 2 is 0.2, the sub-training data volume of the training data corresponding to evaluation model 3 is 2,000, then the score weight corresponding to evaluation model 3 is 0.2, the sub-training data volume of the training data corresponding to evaluation model 4 is 3,000, then the score weight corresponding to evaluation model 4 is 0.3. After determining the score weight corresponding to each evaluation model, the second objective score of the large language model to be tested is calculated according to the weighted sum of the first objective score and the score weight corresponding to each evaluation model. Afterwards, the second objective score and the first manual score are weighted and summed to obtain a global score corresponding to the large language model to be tested.

Furthermore, determining the global score corresponding to the large language model to be tested based on the weighted sum of the first manual score and the second objective score includes:

Performing semantic analysis on the interaction information to be tested to determine target domain information corresponding to the interaction information to be tested;

Based on the target domain information, determining a target manual scoring weight corresponding to the first manual scoring and a target objective scoring weight corresponding to the second objective scoring from a preset mapping relationship; the preset mapping relationship includes a mapping relationship between domain information, manual scoring weights, and objective scoring weights;

A global score corresponding to the large language model to be tested is determined based on the first manual score, the target manual score weight, the second objective score, and the target objective score weight.

Specifically, after determining the first manual score and the second objective score, keyword extraction and semantic analysis can be performed on the interaction information to be tested to determine the target domain information corresponding to the interaction information to be tested. After obtaining the preset mapping relationship, the target domain information is matched with each domain information in the preset mapping relationship, and the manual score weight corresponding to the matched domain information is determined as the target manual score weight corresponding to the first manual score, and the objective score weight corresponding to the matched domain information is determined as the target objective score weight corresponding to the second objective score. After that, the first product of the first manual score and the target manual score weight is calculated, and the second product of the second objective score and the target objective score weight is calculated at the same time. By calculating the sum of the first product and the second product, the global score corresponding to the large language model to be tested can be obtained. After the comprehensive manual score and machine score, the objectivity of the evaluation of the large language model to be tested is improved, thereby improving the reliability of the evaluation result of the large language model to be tested.

It should be noted that due to different work requirements in different fields, when manually scoring the large language model to be tested, the degree of influence of the manual scoring on the global score is different. For example, fields such as the art field and the literature field pay more attention to subjective judgment and experience accumulation, while fields such as the medical field, the science and technology field, and the academic field are more professional and complex, and pay more attention to accuracy. Therefore, if the interactive information to be tested belongs to fields such as the art field and the literature field, the degree of influence of the first manual scoring on the global score is higher than the degree of influence of the second objective scoring on the global score. At this time, the manual scoring weight corresponding to the first manual scoring is greater than the objective scoring weight corresponding to the second objective scoring. If the interactive information to be tested belongs to fields such as the medical field, the science and technology field, and the academic field, the degree of influence of the first manual scoring on the global score is lower than the degree of influence of the second objective scoring on the global score. At this time, the manual scoring weight corresponding to the first manual scoring is less than the objective scoring weight corresponding to the second objective scoring. Therefore, a preset mapping relationship is constructed based on the above content, and the preset mapping relationship includes a mapping relationship between field information, manual scoring weights, and objective scoring weights.

The evaluation method of the large language model provided in the embodiment of the present invention, after obtaining the interaction information to be tested and the first manual score of the user for manually scoring the large language model to be tested, predicts the interaction information to be tested through each evaluation model, outputs the first objective score of each evaluation model for scoring the large language model to be tested, and after combining the first manual score and each first objective score, obtains the global score of the large language model to be tested. After combining the subjective index and the objective index, the objectivity and comprehensiveness of the evaluation of the large language model to be tested can be improved, thereby improving the reliability of the evaluation result of the large language model to be tested. In addition, in the embodiment of the present invention, by determining the target domain information of the interaction information to be tested, the influence of the manual score on the global score is adjusted, and the reliability of the evaluation result is further improved.

The evaluation device for a large language model provided by the present invention is described below. The evaluation device for a large language model described below and the evaluation method for a large language model described above can refer to each other.

The embodiment of the present invention further provides an evaluation device for a large language model. FIG. 2 is a schematic diagram of the structure of the evaluation device for a large language model provided by the embodiment of the present invention. As shown in FIG. 2 , the evaluation device 200 for a large language model includes: an acquisition module 210, an output module 220, and a determination module 230, wherein:

The acquisition module 210 is used to obtain the interaction information to be tested corresponding to the large language model to be tested, and a first manual score of the large language model to be tested by the user based on a manual evaluation template; the manual evaluation template is a scoring template determined based on at least two subjective indicators;

An output module 220 is used to input the interaction information to be tested into at least two evaluation models, and output a first objective score of each evaluation model for scoring the large language model to be tested; the loss function of each evaluation model is determined based on a weighted average of at least two objective indicators;

The determination module 230 is configured to determine a global score corresponding to the large language model to be tested based on the first manual score and each of the first objective scores.

The evaluation device for a large language model provided in an embodiment of the present invention, after obtaining the interaction information to be tested and the first manual score of the user for manually scoring the large language model to be tested, predicts the interaction information to be tested through each evaluation model, outputs the first objective score of each evaluation model for scoring the large language model to be tested, and obtains the global score of the large language model to be tested after combining the first manual score and each first objective score. After combining the subjective index and the objective index, the objectivity and comprehensiveness of the evaluation of the large language model to be tested can be improved, thereby improving the reliability of the evaluation result of the large language model to be tested. In addition, in the embodiment of the present invention, by determining the target domain information of the interaction information to be tested, the influence of the manual score on the global score is adjusted, thereby further improving the reliability of the evaluation result.

Optionally, the acquisition module 210 is specifically configured to:

Obtaining a second subjective score corresponding to each of the subjective indicators by the user;

determining a sum of all second subjective ratings;

Based on the ratio of the sum of all the second subjective scores to the number of all the subjective indicators, a first manual score of the user for scoring the large language model to be tested based on the manual evaluation template is determined.

Optionally, the output module 220 is specifically used for:

For each of the evaluation models, the interaction information to be tested is input into the evaluation model to determine prediction information corresponding to the interaction information to be tested; the prediction information is used to characterize the correctness of the interaction information to be tested;

Based on the prediction information, determining an indicator score corresponding to each of the objective indicators;

Based on the scores of the indicators and the score weights corresponding to the scores of the indicators, a first objective score of the evaluation model for scoring the large language model to be tested is determined.

Optionally, the determination module 230 is specifically configured to:

Obtaining a score weight corresponding to each of the evaluation models, wherein the score weight is determined based on a ratio of an amount of sub-training data corresponding to each of the evaluation models to an amount of total training data corresponding to all the evaluation models;

Determining a second objective score corresponding to the large language model to be tested based on each of the score weights and the first objective score of the evaluation model corresponding to each of the score weights;

A global score corresponding to the large language model to be tested is determined based on a weighted sum of the first manual score and the second objective score.

Optionally, the determination module 230 is specifically configured to:

Performing semantic analysis on the interaction information to be tested to determine target domain information corresponding to the interaction information to be tested;

Based on the target domain information, determining a target manual scoring weight corresponding to the first manual scoring and a target objective scoring weight corresponding to the second objective scoring from a preset mapping relationship; the preset mapping relationship includes a mapping relationship between domain information, manual scoring weights, and objective scoring weights;

A global score corresponding to the large language model to be tested is determined based on the first manual score, the target manual score weight, the second objective score, and the target objective score weight.

Optionally, the objective indicators corresponding to the maximum scoring weights in the loss functions of the evaluation models are different.

Optionally, the large language model evaluation device 200 further includes a construction module, which is specifically used to:

For each of the subjective indicators, determine the number of scoring points corresponding to the subjective indicator;

Based on the number of the scoring scores, determining the degree of the subjective indicator corresponding to each of the scoring scores;

Based on all the scoring scores and the degree of the subjective indicator corresponding to each of the scoring scores, construct a sub-evaluation template corresponding to the subjective indicator;

Constructing the manual evaluation template based on the sub-evaluation templates corresponding to all subjective indicators;

The manual evaluation template and the interaction information to be tested are sent to a terminal corresponding to the user.

FIG3 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. As shown in FIG3 , the electronic device may include: a processor 310, a communication interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the evaluation method of the large language model, and the method includes:

Acquire the interaction information to be tested corresponding to the large language model to be tested, and a first manual score of the large language model to be tested by a user based on a manual evaluation template; the manual evaluation template is a scoring template determined based on at least two subjective indicators;

Inputting the interaction information to be tested into at least two evaluation models, and outputting a first objective score of each evaluation model for scoring the large language model to be tested; the loss function of each evaluation model is determined based on a weighted average of at least two objective indicators;

A global score corresponding to the large language model to be tested is determined based on the first manual score and each of the first objective scores.

In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the large language model evaluation method provided by the above methods, the method includes:

Acquire the interaction information to be tested corresponding to the large language model to be tested, and a first manual score of the large language model to be tested by a user based on a manual evaluation template; the manual evaluation template is a scoring template determined based on at least two subjective indicators;

Inputting the interaction information to be tested into at least two evaluation models, and outputting a first objective score of each evaluation model for scoring the large language model to be tested; the loss function of each evaluation model is determined based on a weighted average of at least two objective indicators;

A global score corresponding to the large language model to be tested is determined based on the first manual score and each of the first objective scores.

In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the large language model evaluation method provided by the above methods, the method comprising:

Acquire the interaction information to be tested corresponding to the large language model to be tested, and a first manual score of the large language model to be tested by a user based on a manual evaluation template; the manual evaluation template is a scoring template determined based on at least two subjective indicators;

Inputting the interaction information to be tested into at least two evaluation models, and outputting a first objective score of each evaluation model for scoring the large language model to be tested; the loss function of each evaluation model is determined based on a weighted average of at least two objective indicators;

A global score corresponding to the large language model to be tested is determined based on the first manual score and each of the first objective scores.

The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM/RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (10)

1. An evaluation method of a large language model, comprising:

Acquiring interaction information to be tested corresponding to a large language model to be tested, and scoring the large language model to be tested by a user based on a manual evaluation template; the manual evaluation template is a scoring template determined based on at least two subjective indexes;

Inputting the interaction information to be tested into at least two evaluation models, and outputting a first objective score of each evaluation model for scoring the large language model to be tested; the loss function of each evaluation model is determined based on a weighted average of at least two objective indicators;

And determining global scores corresponding to the large language model to be tested based on the first artificial scores and the first objective scores.

2. The method for evaluating a large language model according to claim 1, wherein inputting the interaction information to be tested into at least two evaluation models, outputting a first objective score for each evaluation model to score the large language model to be tested, comprises:

inputting the interaction information to be tested into the evaluation model aiming at each evaluation model, and determining prediction information corresponding to the interaction information to be tested; the prediction information is used for representing the correctness of the interaction information to be detected;

determining index scores corresponding to the objective indexes based on the prediction information;

and determining a first objective score of the evaluation model for scoring the large language model to be tested based on each index score and the score weight corresponding to each index score.

3. The method for evaluating a large language model according to claim 1, wherein determining a global score corresponding to the large language model to be tested based on the first artificial score and each of the first objective scores comprises:

Obtaining the score weight corresponding to each evaluation model, wherein the score weight is determined based on the ratio of the sub-training data quantity corresponding to each evaluation model to the total training data quantity corresponding to all the evaluation models;

Determining a second objective score corresponding to the large language model to be tested based on the score weights and the first objective scores of the evaluation models corresponding to the score weights;

And determining a global score corresponding to the large language model to be tested based on the weighted sum of the first artificial score and the second objective score.

4. The method for evaluating a large language model according to claim 3, wherein determining a global score corresponding to the large language model to be tested based on a weighted sum of the first artificial score and the second objective score comprises:

Carrying out semantic analysis on the interaction information to be detected, and determining target field information corresponding to the interaction information to be detected;

Determining a target manual scoring weight corresponding to the first manual scoring and a target objective scoring weight corresponding to the second objective scoring from a preset mapping relation based on the target field information; the preset mapping relation comprises mapping relation among field information, manual scoring weight and objective scoring weight;

And determining global scores corresponding to the large language model to be tested based on the first artificial scores, the target artificial score weights, the second objective scores and the target objective score weights.

5. The method for evaluating a large language model according to any one of claims 1 to 4, wherein objective indicators corresponding to maximum scoring weights in the loss function of each of the evaluation models are different.

6. The method for evaluating a large language model according to any one of claims 1 to 4, wherein obtaining a first artificial score for the user to score the large language model to be evaluated based on an artificial evaluation template comprises:

obtaining second subjective scores corresponding to the subjective indexes by the user;

Determining a sum of all second subjective scores;

And determining a first artificial score of the user for scoring the large language model to be tested based on an artificial evaluation template based on the ratio of the sum of all second subjective scores to the number of all subjective indexes.

7. The method for evaluating a large language model according to any one of claims 1 to 4, further comprising:

Determining the number of scoring values corresponding to the subjective indexes according to the subjective indexes;

determining the degree of the subjective index corresponding to each scoring value based on the number of the scoring values;

constructing a sub-evaluation template corresponding to the subjective index based on all the score values and the degree of the subjective index corresponding to each score value;

Constructing the artificial evaluation template based on the sub-evaluation templates corresponding to all subjective indexes;

and sending the manual evaluation template and the interaction information to be tested to a terminal corresponding to the user.

8. An evaluation device for a large language model, comprising:

The system comprises an acquisition module, a first evaluation module and a second evaluation module, wherein the acquisition module is used for acquiring interaction information to be tested corresponding to a large language model to be tested and a first artificial score for scoring the large language model to be tested based on an artificial evaluation template by a user; the manual evaluation template is a scoring template determined based on at least two subjective indexes;

The output module is used for inputting the interaction information to be tested into at least two evaluation models and outputting a first customer score for scoring the large language model to be tested by each evaluation model; the loss function of each evaluation model is determined based on a weighted average of at least two objective indicators;

and the determining module is used for determining the global score corresponding to the large language model to be tested based on the first artificial score and each first objective score.

9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method for evaluating a large language model according to any one of claims 1-7 when executing the program.

10. A non-transitory computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements a method of evaluating a large language model according to any one of claims 1-7.