CN116956171A

CN116956171A - Classification method, device, equipment and storage medium based on AI model

Info

Publication number: CN116956171A
Application number: CN202211423780.1A
Authority: CN
Inventors: 吴秉哲
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2022-11-14
Filing date: 2022-11-14
Publication date: 2023-10-27

Abstract

A classification method, device, equipment and storage medium based on an AI model relate to the AI technical field. The method comprises the following steps: extracting a characteristic representation of a test sample in the form of text, voice or image through an AI classification model; according to the characteristic representation of the test sample and the sub-group characteristics respectively corresponding to the n sample sub-groups, determining the weight values respectively corresponding to the n sample sub-groups; determining the calibration temperature of the test sample according to the weight values respectively corresponding to the n sample subgroups and the calibration temperatures respectively corresponding to the n sample subgroups; and calibrating the confidence coefficient of the test sample obtained by the AI classification model under a plurality of categories according to the calibration temperature of the test sample to obtain a calibrated classification result. Each sample subgroup has a corresponding calibration temperature, and under the condition of subgroup distribution migration, all sample subgroups can obtain good calibration performance, so that the accuracy of confidence calibration is ensured, and the accuracy of a classification result obtained based on an AI classification model is further improved.

Description

Classification method, device, equipment and storage medium based on AI model

Technical Field

The embodiment of the application relates to the technical field of AI (Artificial Intelligence ), in particular to a classification method, a classification device, classification equipment and a storage medium based on an AI model.

Background

With the development of AI technology, the application of AI models is also increasingly diversified. For example, the AI model may classify samples according to their confidence in multiple categories.

The confidence of samples generated by AI models under multiple categories may be inaccurate, requiring calibration of the confidence of samples under multiple categories. In the related art, a confidence calibration method is provided, a calibration temperature can be configured for an AI model, confidence degrees of samples generated by the AI model under a plurality of categories are adjusted through the calibration temperature, and a final classification result is obtained based on the adjusted confidence degrees.

However, the above method may result in better calibration effect of some subgroups, but worse calibration effect of other subgroups, and further result in worse accuracy of classification result generated by AI model.

Disclosure of Invention

The embodiment of the application provides a classification method, device, equipment and storage medium based on an AI model. The technical scheme provided by the embodiment of the application is as follows.

According to an aspect of an embodiment of the present application, there is provided an AI model-based classification method, including:

extracting a characteristic representation of a test sample in the form of text, voice or image through an AI classification model;

According to the characteristic representation of the test sample, sub-group characteristics respectively corresponding to n sample sub-groups, and determining weight values respectively corresponding to the n sample sub-groups; the method comprises the steps of obtaining a training sample, wherein sub-group characteristics corresponding to an ith sample sub-group in the n sample sub-groups are used for representing average characteristic representation of training samples contained in the ith sample sub-group; the weight value corresponding to the ith sample subgroup is used for representing the possibility that the test sample belongs to the ith sample subgroup, n is an integer greater than 1, and i is a positive integer less than or equal to n;

determining the calibration temperature of the test sample according to the weight values respectively corresponding to the n sample subgroups and the calibration temperatures respectively corresponding to the n sample subgroups; the calibration temperature corresponding to the ith sample subgroup is used for calibrating the classification result of the AI classification model for the ith sample subgroup;

and calibrating the confidence coefficient of the test sample obtained by the AI classification model under a plurality of categories according to the calibration temperature of the test sample to obtain a calibrated classification result, wherein the confidence coefficient is used for representing the prediction probability that the test sample belongs to the category.

According to an aspect of an embodiment of the present application, there is provided an AI model-based classification apparatus, including:

the characteristic extraction module is used for extracting characteristic representations of test samples in text, voice or image forms through the AI model;

the weight determining module is used for determining weight values corresponding to the n sample subgroups according to the characteristic representation of the test sample and the subgroup characteristics corresponding to the n sample subgroups respectively; the method comprises the steps of obtaining a training sample, wherein sub-group characteristics corresponding to an ith sample sub-group in the n sample sub-groups are used for representing average characteristic representation of training samples contained in the ith sample sub-group; the weight value corresponding to the ith sample subgroup is used for representing the possibility that the test sample belongs to the ith sample subgroup, n is an integer greater than 1, and i is a positive integer less than or equal to n;

the temperature determining module is used for determining the calibration temperature of the test sample according to the weight values respectively corresponding to the n sample subgroups and the calibration temperatures respectively corresponding to the n sample subgroups; the calibration temperature corresponding to the ith sample subgroup is used for calibrating the classification result of the AI classification model for the ith sample subgroup;

And the decision calibration module is used for calibrating the confidence coefficient of the test sample obtained by the AI classification model under a plurality of categories according to the calibration temperature of the test sample to obtain a calibrated classification result, wherein the confidence coefficient is used for representing the prediction probability of the test sample belonging to the category.

According to an aspect of an embodiment of the present application, there is provided a computer apparatus including a processor and a memory, in which a computer program is stored, the computer program being loaded and executed by the processor to implement the above-described AI model-based classification method.

According to an aspect of an embodiment of the present application, there is provided a computer-readable storage medium having stored therein a computer program loaded and executed by a processor to implement the above-described AI model-based classification method.

According to an aspect of an embodiment of the present application, there is provided a computer program product comprising a computer program stored in a computer readable storage medium. The processor of the computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program so that the computer device performs the above-described AI-model-based classification method.

The technical scheme provided by the embodiment of the application can bring the following beneficial effects:

and determining the calibration temperature of the test sample through the calibration temperatures respectively corresponding to the n sample subgroups, and calibrating the confidence coefficient of the test sample under a plurality of categories obtained by the AI classification model according to the calibration temperature of the test sample to obtain a calibrated classification result. Each sample subgroup has a corresponding calibration temperature, and under the condition of subgroup distribution migration, all sample subgroups can obtain good calibration performance, so that the accuracy and reliability of confidence calibration are ensured, and the accuracy of a classification result obtained based on an AI classification model is further improved.

Drawings

FIG. 1 is a schematic illustration of an implementation environment for an embodiment of the present application;

FIG. 2 is a schematic diagram of an AI model-based classification method as passed by one embodiment of the application;

FIG. 3 is a flow chart of a classification method based on AI model provided in one embodiment of the application;

FIG. 4 is a flow chart of a classification method based on AI model provided in another embodiment of the application;

FIG. 5 is a schematic diagram of an AI classification model provided in accordance with an embodiment of the application;

FIG. 6 is a flow chart of a medical image classification method based on an AI model provided in one embodiment of the application;

FIG. 7 is a block diagram of an AI model-based classification apparatus provided in accordance with one embodiment of the application;

FIG. 8 is a block diagram of an AI model-based classification apparatus provided in accordance with another embodiment of the application;

fig. 9 is a block diagram of a computer device according to an embodiment of the present application.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the embodiments of the present application will be described in further detail with reference to the accompanying drawings.

Artificial intelligence is the theory, method, technique and application system that uses a digital computer or a digital computer-controlled machine to simulate, extend and expand human intelligence, sense the environment, acquire knowledge and use the knowledge to obtain optimal results. In other words, artificial intelligence is an integrated technology of computer science that attempts to understand the essence of intelligence and to produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence, i.e. research on design principles and implementation methods of various intelligent machines, enables the machines to have functions of sensing, reasoning and decision.

The artificial intelligence technology is a comprehensive subject, and relates to the technology with wide fields, namely the technology with a hardware level and the technology with a software level. Artificial intelligence infrastructure technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and other directions.

Machine Learning (ML) is a multi-domain interdisciplinary, involving multiple disciplines such as probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It is specially studied how a computer simulates or implements learning behavior of a human to acquire new knowledge or skills, and reorganizes existing knowledge structures to continuously improve own performance. Machine learning is the core of artificial intelligence, a fundamental approach to letting computers have intelligence, which is applied throughout various areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, confidence networks, reinforcement learning, transfer learning, induction learning, teaching learning, and the like.

With research and advancement of artificial intelligence technology, research and application of artificial intelligence technology is being developed in various fields, such as common smart home, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned, automatic driving, unmanned aerial vehicles, robots, smart medical treatment, smart customer service, etc., and it is believed that with the development of technology, artificial intelligence technology will be applied in more fields and with increasing importance value.

The scheme provided by the embodiment of the application relates to an artificial intelligence machine learning technology, and is specifically described by the following embodiment.

Before describing embodiments of the present application, some terms involved in the present application will be first described.

1. Subgroup distribution migration: the training sample set is made up of a plurality of sub-groups of samples, and it is assumed that different sub-groups of samples can be obtained during the training process. Common subgroup divisions include gender groups, regional groups in a wind-controlled scene, different hospital groups in a medical AI, and so on. The number of training samples contained in different sub-groups of training samples in the training sample set will often deviate (e.g. more men and less women), and in actual test data we will often assume that the number of training samples contained in different sub-groups of samples is balanced, which is called sub-group distribution migration.

2. Uncertainty estimation (Uncertainty Quantification, UQ): also known as uncertainty quantization. For a model, we are many times unaware of his parameters exactly/accurately, how uncertainty estimation deals with this uncertainty with some Numerical methods.

3. Temperature regulation (Temperature Scaling): temperature scaling is a simple post-processing step that can help the AI model to perform confidence calibration.

Referring to fig. 1, a schematic diagram of an implementation environment of an embodiment of the present application is shown. The solution implementation environment may be implemented as a system architecture that becomes AI model-based classification. The implementation environment of the scheme can comprise: AI model deployment apparatus 100 and method execution apparatus 200.

The AI classification model is used for carrying out AI reasoning and decision on the test sample in the target form to obtain a classification result of the test sample; wherein the target form comprises at least one of: text, speech, images; the classification result of the test sample comprises confidence degrees of the test sample under a plurality of categories, and the confidence degrees are used for representing the prediction probability that the test sample belongs to the category. For the c-th category, the confidence of the test sample under the c-th category refers to the prediction probability that the test sample belongs to the c-th category.

The AI classification model may be a neural network model, such as a CNN (Convolutional Neural Network ) model, RNN (Recurrent Neural Network, recurrent neural network) model, or the like. The AI model deployment device 100 may be the terminal device 101 or the server 102, which is not limited by the present application. The terminal 101 may be an electronic device such as a PC (Personal Computer ), tablet, cell phone, medical device, or the like. The terminal device 101 is deployed with an AI classification model, which may be applied in the risk sensitive field, for example, the AI classification model is applied in the fields of AI medical treatment, AI wind control, automatic driving, etc. Illustratively, the AI classification model is an AI medical model for classification of medical samples.

The server 102 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud computing services. The server 102 may be a server of the terminal device 101 described above for providing services to the terminal device 101.

Communication between the terminal device 101 and the server 102 may be performed through a network, such as a wired or wireless network.

The method execution device 200 can calibrate the classification result generated by the AI classification model, and ensure the accuracy and reliability of the classification result of the AI model.

The method execution device 200 may be the same computer device as the AI model deployment device 100, or may be a different computer device from the AI model deployment device 100, which is not limited in this regard. The computer device refers to an electronic device with data computing, processing and storage capabilities. Illustratively, the AI model deployment device 100 and the method execution device 200 are both terminal devices 101. Illustratively, the AI model deployment device 100 is one of the servers 102 and the method execution device 200 is another server other than the server 102.

According to the classification method based on the AI model, an execution main body of each step can be computer equipment, and the computer equipment refers to electronic equipment with data calculation, processing and storage capacity. Taking the implementation environment of the scheme shown in fig. 1 as an example, the terminal may execute the classification method based on the AI model, the server may execute the classification method based on the AI model, or the terminal and the server may interactively cooperate to execute the classification method based on the AI model, which is not limited in this application. For convenience of explanation, in the following method embodiments, description will be made for a computer device with only the execution subject of each step of the AI model-based classification method.

The confidence of samples obtained by AI models under multiple categories may be inaccurate, and therefore calibration of the classification results of AI models is required. In the related art, a confidence calibration method is provided, which can configure a calibration temperature for an AI model, and calibrate a classification result of a sample obtained by the AI model through the calibration temperature.

However, when there is a case where the sub-group distribution shifts, the sample set is composed of a plurality of sub-groups whose sample number distribution is extremely unbalanced, and thus it may be caused that the AI model exhibits excessive confidence for the sub-group whose sample number is large (i.e., confidence for the sample whose prediction is erroneous is excessively high), and exhibits insufficient confidence for the sample whose sample number is small (i.e., confidence for the sample whose prediction is correct is excessively low). Because the characteristics of the samples contained in each subgroup are different, the same calibration temperature is used for calibrating all the samples, so that the calibration effect of part of subgroups is good, and the calibration effect of other subgroups is poor. Therefore, the embodiment of the application provides a classification method based on an AI model, wherein a calibration temperature is respectively configured for each sample subgroup, the calibration temperature of a test sample is determined according to the calibration temperature of each subgroup, and the classification result of the test sample is determined according to the calibration temperature of the test sample. Therefore, compared with the configuration of only one calibration temperature, each sample subgroup in the scheme of the application has the corresponding calibration temperature, and under the condition of subgroup distribution migration, all sample subgroups can obtain better calibration performance, so that the accuracy and reliability of confidence calibration are ensured, and the accuracy of a classification result obtained based on an AI classification model is further improved.

Optionally, the technical scheme provided by the application can be applied to medical scenes. Illustratively, the AI classification model is an AI medical model for classification of medical samples. For the test medical sample A, the AI medical model outputs a classification result aiming at the test medical sample A, the decision calibration framework can determine a calibration temperature aiming at the test medical sample A, and the confidence coefficient of the test medical sample A under a plurality of categories obtained by the AI medical model is calibrated to obtain a calibrated classification result.

Fig. 2 is a schematic diagram of a classification method based on an AI model according to an embodiment of the application. For one test sample x, the calibration temperature Tx corresponding to the test sample x may be determined according to the calibration temperatures T1, T2, …, tn corresponding to the distribution of the sample subgroups G1, G2, …, gn and the characteristic representation h corresponding to the test sample x obtained by the AI classification model M. And then calibrating the classification result Px of the AI model M aiming at the test sample x according to the calibration temperature Tx corresponding to the test sample x to obtain a calibrated classification result P.

Referring to fig. 3, a flowchart of a classification method based on an AI model according to an embodiment of the application is shown. The method may include at least one of the following steps 210-240.

At step 210, feature representations of test samples in text, speech, or image form are extracted by the AI classification model.

The test sample refers to a sample to be predicted.

The feature representation of the test sample refers to the hidden layer features of the test sample. In some embodiments, the features of the test sample are represented as a feature vector. In some embodiments, the feature representation of the test sample is a high-dimensional feature vector. The characteristic representation of the test sample is used to further generate a classification result for the test sample. Illustratively, the AI model includes an input layer, a hidden layer, and an output layer. The AI model may include at least one hidden layer. The feature representation of the test sample may be a feature vector obtained by any hidden layer of the AI model, which is not limited in this regard by the present application.

Step 220, determining weight values corresponding to the n sub-groups of samples according to the characteristic representation of the test sample and the sub-group characteristics corresponding to the n sub-groups of samples respectively; the method comprises the steps of determining a training sample, wherein sub-group characteristics corresponding to an ith sample sub-group in n sample sub-groups are used for representing average characteristic representation of training samples contained in the ith sample sub-group; and the weight value corresponding to the ith sample subgroup is used for representing the possibility that the test sample belongs to the ith sample subgroup, n is an integer greater than 1, and i is a positive integer less than or equal to n.

In some embodiments, the feature representation of the training sample refers to a feature representation of the training sample extracted by the AI classification model. The feature of the training sample is expressed as a feature vector, and the dimensions of the feature expression corresponding to each training sample are the same. The average feature representation of the training samples contained in the ith sample subgroup refers to the average of the feature representations of the training samples contained in the ith sample subgroup. The average value of the feature vectors means an average feature vector formed by the average value of the position elements corresponding to the respective feature vectors.

In some embodiments, the weight values respectively corresponding to the n sub-groups of samples may be determined according to the similarity between the feature representation of the test sample and the sub-group features respectively corresponding to the n sub-groups of samples. The application is not limited to a method for calculating the similarity between the characteristic representations of the test samples and the sub-group characteristics corresponding to the n sub-groups of samples, respectively. Such as cosine similarity, pearson correlation coefficient, manhattan distance, euclidean distance, etc.

Illustratively, the weight values respectively corresponding to the n sub-groups of samples are determined according to the distance values between the feature representations of the test samples and the sub-features respectively corresponding to the n sub-groups of samples. For example, the weight values corresponding to the n sub-groups of samples are determined from euclidean distance values between the features of the test sample representing the sub-features corresponding to the n sub-groups of samples, respectively.

In some embodiments, distance values between characteristic representations of the test samples and subgroup characteristics corresponding to each sample subgroup are calculated respectively, so that distance values corresponding to n sample subgroups respectively are obtained; the distance value corresponding to the ith sample subgroup is used for representing the proximity between the characteristic representation of the test sample and the subgroup characteristic corresponding to the ith sample subgroup; and determining the weight values corresponding to the n sample subgroups according to the distance values corresponding to the n sample subgroups respectively.

The method for determining the weight values corresponding to the n sample subgroups according to the distance values corresponding to the n sample subgroups is not limited in the present application.

Illustratively, the weight value corresponding to the ith sample subgroup is determined according to the sum of the distance values corresponding to the ith sample subgroup and the distance values corresponding to the n sample subgroups.

For example, according to the distance value corresponding to the ith sample subgroup, the weight value corresponding to the ith sample subgroup is determined according to the ratio between the sum of the distance values corresponding to the n sample subgroups.

In some embodiments, as shown in fig. 4, before step 220, the method further includes step 212 of dividing the training samples included in the training sample set into n sub-groups of samples; respectively extracting characteristic representations of each training sample through an AI classification model; and for the ith sample subgroup in the n sample subgroups, averaging the feature representations of the training samples contained in the ith sample subgroup to obtain subgroup features corresponding to the ith sample subgroup.

In some embodiments, the number of training samples included in the n sample subsets may be the same or different, which is not limited by the present application. For example, the number of training samples included in the n sample subsets is the same as 100. For example, the number of training samples contained in n sample subsets is different, wherein the number of training samples contained in 1 sample subset is 200 and the number of training samples contained in another sample subset is 20.

In some embodiments, the AI classification model in step 212 described above is a trained AI classification model. The application is not limited to samples that train the AI classification model. For example, the AI classification model may be trained by training a sample set to obtain a trained AI classification model, and then step 212 is performed based on the trained AI classification model. For example, the AI classification model may be trained by samples independent of the training sample set, resulting in a trained AI classification model, and then step 212 is performed in accordance with the trained AI classification model.

Step 230, determining a calibration temperature of the test sample according to the weight values respectively corresponding to the n sample subgroups and the calibration temperatures respectively corresponding to the n sample subgroups; and the calibration temperature corresponding to the ith sample subgroup is used for calibrating the classification result of the AI classification model for the ith sample subgroup.

The calibration temperature is a parameter used for calibrating a classification result generated by the AI classification model, and more specifically, the calibration temperature is used for adjusting the confidence coefficient of a sample output by the AI classification model under each category, so that the accuracy of the confidence coefficient is improved. The temperature adjustment technique described above is used, and is therefore referred to as "calibration temperature".

The classification result of the ith sample subgroup refers to classification results respectively corresponding to training samples contained in the ith sample subgroup, and the classification results comprise confidence degrees under a plurality of categories respectively corresponding to the training samples contained in the ith sample subgroup, wherein the confidence degrees are used for representing the prediction probability that the training samples belong to the categories.

The application is not limited with respect to the method of determining the calibration temperature of the test sample. For example, according to the weight values corresponding to the n sample subgroups, the calibration temperatures corresponding to the n sample subgroups are weighted and summed to obtain the calibration temperature of the test sample. For example, according to the weight values corresponding to the n sample subgroups, a weighted average of the calibration temperatures corresponding to the n sample subgroups is calculated, so as to obtain the calibration temperature of the test sample.

In some embodiments, as shown in fig. 4, before step 230, the method further includes step 222, performing subgroup division on the verification samples included in the verification sample set to obtain n groups of verification samples; wherein the ith group of verification samples in the n groups of verification samples comprises: at least one verification sample belonging to the same subgroup as the training sample in the ith sample subgroup; and for the ith group of verification samples, acquiring a calibration temperature which enables the calibration error of the AI classification model on the ith group of verification samples to meet the condition, and taking the calibration temperature as the calibration temperature corresponding to the ith sample subgroup.

The number of verification samples included in the verification sample set may be the same as or different from the number of training samples included in the training sample set. For example, the number of verification samples included in the verification sample set and the number of training samples included in the training sample set are 500. For example, the verification sample set contains 500 verification samples and the training sample set contains 1000 training samples.

In some embodiments, the n sets of validation samples each contain the same number of validation samples. For example, the verification sample set contains 500 verification samples, and the verification samples contained in the verification sample set are subjected to subgroup division to obtain 10 groups of verification samples, wherein the number of the verification samples contained in each group of verification samples is 50.

In some embodiments, the calibration error is used to quantitatively characterize the accuracy of the decision uncertainty characterization.

In some embodiments, the calibration temperature that causes the calibration error of the AI classification model on the ith set of verification samples to satisfy the condition is obtained by an iterative method as the calibration temperature corresponding to the ith sub-group of samples.

The application is not limited with respect to specifically calibrating errors. For example, ECE (Expected Calibration Error, average calibration error) may be employed as the calibration error. Illustratively, a negative log likelihood may be employed as the calibration error.

In some embodiments, the calibration error satisfaction condition includes at least one of: the calibration error is minimized and the maximum number of iterations is reached.

And step 240, calibrating the confidence coefficient of the test sample obtained by the AI classification model under a plurality of categories according to the calibration temperature of the test sample, and obtaining a calibrated classification result.

In some embodiments, a feature mapping result corresponding to the feature representation of the test sample is obtained through a fully connected layer of the AI classification model; and determining the confidence coefficient of the test sample after calibration under a plurality of categories according to the ratio of the feature mapping result to the calibration temperature of the test sample, and obtaining the calibrated classification result.

In some embodiments, the AI model includes a feature extraction layer and a decision layer, which includes a full connection layer and a classification layer (or classifier). The feature extraction layer is used for extracting a feature representation of the test sample. The full connection layer is used for obtaining feature mapping results corresponding to the feature representation of the test sample. The classification layer is used for determining the confidence coefficient of the test sample after calibration under a plurality of categories according to the ratio of the characteristic mapping result to the calibration temperature of the test sample.

According to the technical scheme provided by the embodiment of the application, the calibration temperature of the test sample is determined through the calibration temperatures respectively corresponding to the n sample subgroups, and the confidence degrees of the test sample obtained by the AI classification model under a plurality of categories are calibrated according to the calibration temperature of the test sample, so that a calibrated classification result is obtained. Each sample subgroup has a corresponding calibration temperature, and under the condition of subgroup distribution migration, all sample subgroups can obtain good calibration performance, so that the accuracy and reliability of confidence calibration are ensured, and the accuracy of a classification result obtained based on an AI classification model is further improved.

Referring to fig. 4, a flowchart of a classification method based on an AI model according to an embodiment of the application is shown. The method may include at least one of the following steps 210-240.

In some embodiments, the AI classification model is a trained AI classification model.

In some embodiments, the AI classification model includes a feature extraction layer through which a feature representation of the test sample is extracted.

In some embodiments, a characterization of the test sample is extracted by the following formula.

h＝f(x；θ)

Where h refers to the characteristic representation of the test sample, x refers to the test sample, and θ refers to the parameters of the AI classification model.

Step 212, dividing the training samples contained in the training sample set into n sample subgroups; respectively extracting characteristic representations of each training sample through an AI classification model; and for the ith sample subgroup in the n sample subgroups, averaging the feature representations of the training samples contained in the ith sample subgroup to obtain subgroup features corresponding to the ith sample subgroup.

In some embodiments, the number of training samples contained in the n sub-groups of samples varies. For example, the number of training samples included in the ith sample subgroup is G _i 。

In some embodiments, feature representations of the respective training samples are extracted separately by a feature extraction layer.

In some embodiments, an average value of the feature representations of the training samples included in the ith sample subgroup is taken as a subgroup feature corresponding to the ith sample subgroup.

Illustratively, the subgroup characteristics corresponding to the ith sample subgroup are determined by the following formula.

Wherein g _i Refers to subgroup features corresponding to the ith sample subgroup, |G _i I refers to the number of training samples contained in the ith subgroup of samples, x refers to the training samples contained in the ith subgroup of samples, f (x)The method comprises the steps of carrying out a first treatment on the surface of the θ) refers to the feature representation of training sample x.

It should be noted that step 212 may be performed before step 210 or after step 210, which is not limited by the present application.

The application is not limited to a particular method of calculating distance values between sub-group features of the test sample that represent sub-groups of samples. For example, euclidean distances (also referred to as euclidean distances) between feature representations of the test samples and subgroup features corresponding to respective subgroups of samples may be calculated, as may manhattan distances between feature representations of the test samples and subgroup features corresponding to respective subgroups of samples.

For example, euclidean distance values between the feature representations of the test samples and the subgroup features corresponding to the respective sample subgroups are calculated, respectively, to obtain distance values corresponding to the n sample subgroups, respectively.

For example, the euclidean distance value between the feature representation of the test sample and the subgroup feature corresponding to the ith sample subgroup is calculated by the following formula, to obtain the distance value corresponding to the ith sample subgroup.

d _i ＝|｜h-g _i || ²

Wherein d _i Refers to the distance value corresponding to the ith sample subgroup, h refers to the characteristic representation of the test sample, g _i Refers to subgroup features corresponding to the ith sample subgroup, and refers to I h-g _i || ² The Euclidean distance value between the feature representation of the test sample and the subgroup feature corresponding to the ith subgroup of samples is calculated.

In some embodiments, the weight value corresponding to the ith sub-group of samples is determined according to the sum of the distance values corresponding to the ith sub-group of samples and the distance values corresponding to the n sub-groups of samples, respectively.

For example, the weight value corresponding to the ith sample subgroup is determined by the following formula.

w _i ＝d _i /∑d _i

Wherein w is _i Refers to the weight value, d, corresponding to the ith sample subgroup _i Refers to the distance value corresponding to the ith sample subgroup, sigma d _i Refers to the sum of distance values corresponding to the n sample subgroups respectively.

Step 222, performing subgroup division on verification samples contained in the verification sample set to obtain n groups of verification samples; wherein the ith group of verification samples in the n groups of verification samples comprises: at least one verification sample belonging to the same subgroup as the training sample in the ith sample subgroup; and for the ith group of verification samples, acquiring a calibration temperature which enables the calibration error of the AI model on the ith group of verification samples to meet the condition, and taking the calibration temperature as the calibration temperature corresponding to the ith sample subgroup.

In some embodiments, the n sets of validation samples each contain the same number of validation samples.

In some embodiments, the verification samples included in the ith set of verification samples may be divided into the ith sub-group of samples.

In some examples, an initialized calibration temperature is obtained; obtaining classification results respectively corresponding to all verification samples contained in the ith group of verification samples through an AI classification model; in the kth iteration process, calibrating the classification result corresponding to the ith group of verification samples according to the calibration temperature of the kth iteration to obtain the calibrated classification result corresponding to the ith group of verification samples; wherein k is a positive integer, the initial value of k is 1, and the calibration dimension of the 1 st iteration is the initialized calibration temperature; determining a calibration error corresponding to the ith group of verification samples according to the calibrated classification result corresponding to the ith group of verification samples and the label information corresponding to the ith group of verification samples; adjusting the calibration temperature of the kth iteration by taking the calibration error corresponding to the minimized ith group of verification samples as a target to obtain the calibration temperature of the kth+1th iteration; if the calibration error corresponding to the ith group of verification samples does not meet the condition, k=k+1, and calibrating the classification result corresponding to the ith group of verification samples again from the calibration temperature according to the kth iteration to obtain the calibrated classification result corresponding to the ith group of verification samples; and if the calibration error corresponding to the ith group of verification samples meets the condition, determining the calibration temperature of the (k+1) th iteration as the calibration temperature corresponding to the ith sample subgroup.

The application is not limited to the value of the initialized calibration temperature. For example, an initialized calibration temperature T ₀ ＝2。

In some embodiments, the AI classification model includes a feature extraction layer and a decision layer, where feature representations respectively corresponding to each verification sample included in the ith group of verification samples are obtained by the feature extraction layer, and classification results respectively corresponding to each verification sample included in the ith group of verification samples are obtained by the decision layer.

In some embodiments, the decision layer includes a full connection layer and a classification layer. The full connection layer is used for determining feature mapping results (logits) respectively corresponding to all the verification samples contained in the ith group of verification samples according to the feature representations respectively corresponding to all the verification samples contained in the ith group of verification samples; the classification layer is used for determining classification results respectively corresponding to all verification samples contained in the ith group of verification samples according to feature mapping results (logits) respectively corresponding to all verification samples contained in the ith group of verification samples.

Illustratively, as shown in fig. 5, the AI classification model 400 includes a feature extraction layer 410 and a decision layer 420, the decision layer 420 including a full connection layer 421 and a classification layer 422.

In some embodiments, feature mapping results (logits) respectively corresponding to each verification sample included in the ith set of verification samples are determined by the following formula.

z＝W·h

Wherein z refers to a feature mapping result (logits) corresponding to the verification sample, W refers to a decision layer parameter matrix, and h refers to a feature representation corresponding to the verification sample.

In some embodiments, a classification result corresponding to each verification sample included in the ith set of verification samples is determined by a classifier. The present application is not limited to the kind of classifier. For example, a Softmax classifier may be selected.

Illustratively, the classification results respectively corresponding to the respective verification samples included in the i-th group of verification samples are determined by the following formula.

Wherein p is _i Means verifying the confidence of the sample under the ith category, C means the number of categories, z _i Refers to verifying the feature mapping result (logits) of the sample corresponding to the ith category.

In some embodiments, verifying the confidence of the sample refers to verifying the maximum value of the confidence of the sample under each category, respectively. Illustratively, conf=maxp _i I=1, 2, …, C, where conf refers to the confidence of the validation sample, p _i Refers to verifying the confidence of the sample under the ith category, and C refers to the number of categories.

In some embodiments, the classification result corresponding to the ith set of verification samples is calibrated according to the calibration temperature of the kth iteration, so as to obtain a calibrated classification result corresponding to the ith set of verification samples.

Illustratively, according to the characteristic mapping results (logits) respectively corresponding to the calibration temperature of the kth iteration and each verification sample included in the ith group of verification samples, calibrating the classification result corresponding to the ith group of verification samples to obtain a calibrated classification result corresponding to the ith group of verification samples.

For example, according to the ratio of the calibration temperature of the kth iteration to the feature mapping results (logits) respectively corresponding to each verification sample included in the ith group of verification samples, calibrating the decision result corresponding to the ith group of verification samples to obtain the calibrated classification result corresponding to the ith group of verification samples.

For example, the classification result corresponding to the ith group of verification samples is calibrated through the following formula, so as to obtain the calibrated classification result corresponding to the ith group of verification samples.

Wherein p is _i Means verifying the confidence of the sample after calibration under the ith category, T means the calibration temperature of the kth iteration, C means the number of categories, z _i Refers to verifying the feature mapping result (logits) of the sample corresponding to the ith category. At this time, the value of T cannot be 0.

In some embodiments, the calibration error corresponding to the ith set of verification samples is determined according to the calibrated classification result corresponding to the ith set of verification samples and the label information corresponding to the ith set of verification samples.

The application is not limited with respect to specifically calibrating errors. For example, ECE may be employed as the calibration error. Illustratively, a negative log likelihood may be employed as the calibration error.

In some embodiments, ECE is employed as the calibration error. In some embodiments, the calibration error corresponding to the ith set of verification samples is determined according to the calibrated classification result corresponding to the ith set of verification samples and the label information corresponding to the ith set of verification samples. Illustratively, the accuracy corresponding to the ith group of verification samples is determined according to the label information corresponding to the ith group of verification samples, and the calibration error corresponding to the ith group of verification samples is determined according to the accuracy corresponding to the ith group of verification samples and the calibrated classification result corresponding to the ith group of verification samples.

In some embodiments, according to the calibrated classification result corresponding to the ith group of verification samples, the ith group of verification samples are divided into M sample intervals, and according to the accuracy rates respectively corresponding to the M sample intervals and the calibrated classification result corresponding to the verification sample distribution contained in the M sample intervals, the calibration error corresponding to the ith group of verification samples is determined.

Illustratively, the validation sample is partitioned into M different sample intervals (bins) according to the confidence of the validation sample (where the mth sample interval is denoted as B _m ) And determining a calibration error according to the accuracy of the verification samples corresponding to the M sample intervals and the average value of the confidence coefficient of the verification samples contained in the sample intervals.

The method for dividing the M sample intervals is not limited in the present application.

Illustratively, the accuracy of the validation samples contained within the mth sample interval is denoted as acc (B _m ) The average value of the confidence of the verification sample contained in the mth sample interval is expressed as conf (B _m ) The calibration error ECE is calculated by the following formula.

Where ECE refers to the calibration error and n represents the number of validation samples contained in the set of validation samples.

The smaller ECE, the more accurate the decision result of the AI classification model is calibrated, and the higher the reliability of the AI classification model is. At this time, the calibration error satisfying the condition means ECE minimization.

In some embodiments, a negative log likelihood is employed as the calibration error. Illustratively, the negative log likelihood is calculated by the following formula.

Wherein L is negative log likelihood, V is a set of verification samples, x is a verification sample, p _y Refers to the confidence of x over category y.

In some embodiments, the calibration temperature for the k+1th iteration is updated by a gradient update method.

The method is used for determining the calibration temperature corresponding to the ith sample subgroup and ensuring the accuracy of the calibration temperature corresponding to the ith sample subgroup.

In some embodiments, a calibration temperature range corresponding to the ith sample subgroup is set, calibration temperatures contained in the calibration temperature range are traversed through a grid search method in the calibration temperature range corresponding to the ith sample subgroup, and the calibration temperature corresponding to the ith sample subgroup is determined.

In some embodiments, the calibration temperature ranges corresponding to the n sample subgroups may be the same or different, which is not limited in the present application.

In some embodiments, the value of the calibration temperature range corresponding to the ith sample subgroup is not limited to the present application, and may be set according to specific implementation. At this time, the calibration error satisfying the condition means that the maximum number of iterations is reached. By setting the calibration temperature range, grid search is performed in the calibration temperature range to determine the calibration temperature corresponding to the ith sample subgroup, and the calculation amount required for determining the calibration temperature corresponding to the ith sample subgroup is reduced.

It should be noted that, step 222 may be performed before step 220 or after step 220, which is not limited in the present application.

Step 230, determining a calibration temperature of the test sample according to the weight values respectively corresponding to the n sample subgroups and the calibration temperatures respectively corresponding to the n sample subgroups; and the calibration temperature corresponding to the ith sample subgroup is used for calibrating the classification result of the AI model for the ith sample subgroup.

In some embodiments, according to the weight values corresponding to the n sub-groups of samples, the calibration temperatures corresponding to the n sub-groups of samples are weighted and summed to obtain the calibration temperature of the test sample.

Illustratively, the calibration temperature of the test sample is determined according to the following formula.

T＝∑w _i T _i

Wherein T is the calibration temperature of the test sample, w _i Refers to the weight value corresponding to the ith sample subgroup, T _i Refers to the calibration temperature corresponding to the ith sample subgroup.

The calibration temperature corresponding to the test sample is determined according to the weight value corresponding to each sample subgroup and the calibration temperature corresponding to each sample subgroup, so that the characteristics of the test sample are considered, the characteristics corresponding to each sample subgroup are considered, and the problems of over confidence and under confidence in the sample distribution migration condition can be well avoided.

In some embodiments, the confidence level of the test sample obtained by the AI classification model under a plurality of categories is calibrated according to the ratio of the calibration temperature of the test sample to the feature mapping result (logits) of the AI classification model for the test sample, so as to obtain a calibrated classification result.

For example, the confidence degrees of the test samples obtained by the AI classification model under a plurality of categories are calibrated through the following formula, and the calibrated classification result is obtained.

Wherein p is _i The confidence of the test sample after calibration under the ith category is shown as T, the calibration temperature of the test sample is shown as C, the number of categories is shown as z _i Refers to the feature mapping result (logits) of the test sample corresponding to the ith category. At this time, the value of T cannot be 0.

According to the technical scheme provided by the embodiment of the application, the corresponding calibration temperatures are respectively configured for the n sample subgroups, and the corresponding calibration temperatures of the test samples are determined according to the corresponding calibration temperatures of the sample subgroups, so that the classification results of the test samples are calibrated. Because each sample subgroup has corresponding calibration temperature, a good calibration effect can be obtained under the condition of subgroup distribution migration, and the accuracy and reliability of the classification result of the AI classification model are ensured.

The AI classification model is used for carrying out AI reasoning and decision on the test sample in the target form to obtain a decision result of the test sample; wherein the target form comprises at least one of: text, speech, images; the decision result of the test sample includes confidence levels of the test sample under a plurality of categories, the confidence levels being used to characterize a predictive probability that the test sample belongs to a category.

Taking an image as an example, an AI classification model is used for reasoning to obtain a classification result (such as fracture, perfection, osteoporosis, etc.) corresponding to the medical image according to the medical image, and evaluating the classification result corresponding to the medical image, namely evaluating the confidence of the classification result corresponding to the medical image. If the confidence coefficient of the classification result corresponding to the medical image is lower than a threshold value, introducing a manual intervention model to avoid medical accidents.

For example, the test sample is a medical image to be predicted and the subgroup of samples is a subgroup of medical image samples. As shown in fig. 6, the AI model-based medical image classification method includes at least one of the following steps 610-640.

At step 610, a feature representation of the medical image to be predicted is extracted by the AI classification model.

In some embodiments, the feature representation of the medical image to be predicted is extracted by the following formula.

h＝f(x；θ)

Where h refers to the feature representation of the medical image to be predicted, x refers to the medical image to be predicted, and θ refers to the parameters of the AI classification model.

Step 620, determining weight values corresponding to the n sub-groups of medical image samples according to the feature representation of the medical image to be predicted and the sub-group features corresponding to the n sub-groups of medical image samples, respectively; the method comprises the steps of determining a subgroup characteristic corresponding to an ith medical image sample subgroup in n medical image sample subgroups, wherein the subgroup characteristic corresponding to the ith medical image sample subgroup is used for representing an average characteristic representation of training medical image samples contained in the ith medical image sample subgroup; the weight value corresponding to the ith medical image sample subgroup is used for representing the possibility that the test medical image sample belongs to the ith medical image sample subgroup, n is an integer greater than 1, and i is a positive integer less than or equal to n.

In some embodiments, step 620 further includes step 612 of dividing the training samples included in the training sample set into n sub-groups of medical image samples; respectively extracting characteristic representations of each training sample through an AI classification model; and for the ith medical image sample subgroup in the n medical image sample subgroups, averaging the feature representations of the training samples contained in the ith medical image sample subgroup to obtain subgroup features corresponding to the ith medical image sample subgroup.

In some embodiments, the subgroup features corresponding to the ith medical image sample subgroup are determined by the following formula.

Wherein g _i Refers to subgroup features corresponding to the ith medical image sample subgroup, |G _i The term "x" refers to the number of training samples contained in the i-th medical image sample subgroup, x refers to the training samples contained in the i-th medical image sample subgroup, and f (x; θ) refers to the feature representation of training sample x.

In some embodiments, the euclidean distance value between the feature representation of the medical image to be predicted and the subgroup feature corresponding to the ith medical image sample subgroup is calculated by the following formula, resulting in the distance value corresponding to the ith medical image sample subgroup.

d _i ＝||h-g _i || ²

Wherein d _i Refers to the distance value corresponding to the ith medical image sample subgroup, h refers to the characteristic representation of the medical image to be predicted, g _i Refers to subgroup features corresponding to the ith medical image sample subgroup, and refers to I h-g _i || ² A euclidean distance value between feature representations of the medical image to be predicted and subgroup features corresponding to the ith medical image sample subgroup is calculated.

In some embodiments, the weight value corresponding to the ith sub-group of samples is determined from the sum of the distance values corresponding to the ith sub-group of medical image samples and the distance values corresponding to the n sub-groups of medical image samples, respectively.

For example, the weight value corresponding to the ith medical image sample subgroup is determined according to the ratio between the distance value corresponding to the ith medical image sample subgroup and the sum of the distance values respectively corresponding to the n medical image sample subgroups.

For example, the weight value corresponding to the ith medical image sample subgroup is determined by the following formula.

w _i ＝d _i /∑d _i

Wherein w is _i Refers to the weight value d corresponding to the ith medical image sample subgroup _i Refers to the distance value corresponding to the ith medical image sample subgroup, sigma d _i Refers to the sum of distance values respectively corresponding to n medical image sample subgroups.

Step 630, determining a calibration temperature of the medical image to be predicted according to the weight values respectively corresponding to the n medical image sample subgroups and the calibration temperatures respectively corresponding to the n medical image sample subgroups; the calibration temperature corresponding to the ith medical image sample subgroup is used for calibrating the classification result of the AI classification model for the ith medical image sample subgroup.

In some embodiments, step 630 is preceded by step 622 of performing subgroup division on the verification samples included in the verification sample set to obtain n groups of verification samples; wherein the ith group of verification samples in the n groups of verification samples comprises: at least one verification sample belonging to the same subgroup as the training sample in the ith subgroup of medical image samples; and for the ith group of verification samples, acquiring a calibration temperature which enables the calibration error of the AI classification model on the ith group of verification samples to meet the condition, and taking the calibration temperature as the calibration temperature corresponding to the ith medical image sample subgroup.

In some examples, an initialized calibration temperature is obtained; obtaining classification results respectively corresponding to all verification samples contained in the ith group of verification samples through an AI classification model; in the kth iteration process, calibrating the classification result corresponding to the ith group of verification samples according to the calibration temperature of the kth iteration to obtain the calibrated classification result corresponding to the ith group of verification samples; wherein k is a positive integer, the initial value of k is 1, and the calibration dimension of the 1 st iteration is the initialized calibration temperature; determining a calibration error corresponding to the ith group of verification samples according to the calibrated classification result corresponding to the ith group of verification samples and the label information corresponding to the ith group of verification samples; adjusting the calibration temperature of the kth iteration by taking the calibration error corresponding to the minimized ith group of verification samples as a target to obtain the calibration temperature of the kth+1th iteration; if the calibration error corresponding to the ith group of verification samples does not meet the condition, k=k+1, and calibrating the classification result corresponding to the ith group of verification samples again from the calibration temperature according to the kth iteration to obtain the calibrated classification result corresponding to the ith group of verification samples; and if the calibration error corresponding to the ith group of verification samples meets the condition, determining the calibration temperature of the (k+1) th iteration as the calibration temperature corresponding to the ith medical image sample subgroup.

And step 640, calibrating the confidence coefficient of the medical image to be predicted, which is obtained by the AI classification model, under a plurality of categories according to the calibration temperature of the medical image to be predicted, so as to obtain a calibrated classification result, wherein the confidence coefficient is used for representing the prediction probability that the medical image to be predicted belongs to the category.

In some embodiments, the confidence level of the medical image to be predicted, which is obtained by the AI classification model, under a plurality of categories is calibrated by the following formula, and a calibrated classification result is obtained.

Wherein p is _i Refers to the confidence of the medical image to be predicted after calibration under the ith category, T refers to the calibration temperature of the medical image to be predicted, C refers to the number of categories, z _i Refers to the feature mapping result (logits) of the medical image to be predicted corresponding to the ith category. At this time, the value of T cannot be 0.

Because the confidence coefficient generated by the AI classification model is often inaccurate, the confidence coefficient of a certain wrong decision is too high, and the reliability degree of the whole system is affected. The number of medical images corresponding to different classification results generally differs greatly, for example, the number of medical images in rare categories (such as Ma Fanzeng syndrome) is very small, while the number of medical images in common categories (such as lumbar disc herniation) is very large, so that obvious sub-group distribution migration conditions exist. It is therefore easy for the AI classification model to exhibit excessive confidence (i.e., too high a confidence in samples of mispredictions) for subgroups where medical images of common categories are located, and to exhibit insufficient confidence (i.e., too low a confidence in samples of correctly predicted) for subgroups where medical images of rare categories are located. Due to inaccurate confidence, a manual intervention model cannot be introduced in time, so that a medical accident occurs.

According to the classification scheme based on the AI model, the AI classification model is calibrated according to the classification result corresponding to the medical image obtained by the medical image, and because the corresponding calibration temperature is configured for each subgroup, the problems can be avoided to a great extent, and when the confidence level of the classification result corresponding to the medical image is lower than the threshold value, the manual intervention model can be introduced in time, so that medical accidents are avoided.

In the above embodiment, only the medical image is taken as an example, and the technical solution provided in the embodiment of the present application may also be applied to a classification method for other images, for example, the automatic driving field classifies the driving state of the vehicle (for example, straight, left turn, right turn, etc.) according to the collected driving image of the vehicle, and the tag recognition field classifies the image tag (for example, barcode, two-dimensional code, etc.). Of course, the technical scheme provided by the embodiment of the application can also be applied to a text classification method and a voice classification method. Illustratively, the scheme of the application can be applied to the field of semantic recognition. For example, classifying authored forms of text (e.g., prose, novels, poems, etc.) according to article content; as another example, topics for the articles are categorized according to the content of the articles (e.g., the articles are entitled to work, school, etc.). The application is also applicable, by way of example, to the field of audio separation technology. For example, separating song audio with accompaniment into vocal audio and accompaniment audio; for another example, audio is classified into noisy audio and non-noisy audio. The technical scheme provided by the embodiment of the application is not limited to the application scene, and any scene for classifying the sample based on the AI classification model can be suitable for the scheme of the application.

The following are examples of the apparatus of the present application that may be used to perform the method embodiments of the present application. For details not disclosed in the embodiments of the apparatus of the present application, please refer to the embodiments of the method of the present application.

Referring to fig. 7, a block diagram of an AI model-based classification apparatus according to an embodiment of the application is shown. The device has the function of realizing the method example, and the function can be realized by hardware or can be realized by executing corresponding software by hardware. The apparatus may be the computer device described above or may be provided in a computer device. As shown in fig. 7, the apparatus 700 includes: feature extraction module 710, weight determination module 720, temperature determination module 730, and decision calibration module 740.

The feature extraction module 710 is configured to extract a feature representation of a test sample in the form of text, speech, or image through an AI classification model.

The weight determining module 720 is configured to determine weight values corresponding to the n sub-groups of samples according to the feature representation of the test sample, where the feature representation corresponds to the sub-group feature of the n sub-groups of samples; the method comprises the steps of obtaining a training sample, wherein sub-group characteristics corresponding to an ith sample sub-group in the n sample sub-groups are used for representing average characteristic representation of training samples contained in the ith sample sub-group; and the weight value corresponding to the ith sample subgroup is used for representing the possibility that the test sample belongs to the ith sample subgroup, n is an integer greater than 1, and i is a positive integer less than or equal to n.

A temperature determining module 730, configured to determine a calibration temperature of the test sample according to the weight values respectively corresponding to the n sample subgroups and the calibration temperatures respectively corresponding to the n sample subgroups; and the calibration temperature corresponding to the ith sample subgroup is used for calibrating the classification result of the AI classification model for the ith sample subgroup.

And a result calibration module 740, configured to calibrate, according to the calibration temperature of the test sample, confidence degrees of the test sample obtained by the AI classification model under a plurality of categories, to obtain a calibrated classification result, where the confidence degrees are used to characterize a prediction probability that the test sample belongs to the category.

In some embodiments, the weight determining module 720 is configured to calculate distance values between the feature representations of the test samples and the subgroup features corresponding to each of the sample subgroups, so as to obtain distance values corresponding to the n sample subgroups respectively; the distance value corresponding to the ith sample subgroup is used for representing the proximity between the characteristic representation of the test sample and the subgroup characteristic corresponding to the ith sample subgroup; and determining the weight values corresponding to the n sample subgroups according to the distance values corresponding to the n sample subgroups respectively.

In some embodiments, the temperature determining module 730 is configured to perform weighted summation on the calibration temperatures corresponding to the n sub-groups of samples according to the weight values corresponding to the n sub-groups of samples, so as to obtain the calibration temperature of the test sample.

In some embodiments, as shown in fig. 7, the apparatus 700 further comprises a subgroup feature extraction module 750.

A subgroup feature extraction module 750, configured to divide the training samples included in the training sample set into the n sample subgroups; respectively extracting characteristic representations of the training samples through the AI classification model; and for the ith sample subgroup in the n sample subgroups, averaging the feature representations of the training samples contained in the ith sample subgroup to obtain subgroup features corresponding to the ith sample subgroup.

In some embodiments, as shown in fig. 7, the apparatus 700 further includes a sub-group temperature determination module 760.

The subgroup temperature determining module 760 is configured to subgroup the verification samples included in the verification sample set, so as to obtain n groups of verification samples; wherein an ith set of validation samples of the n sets of validation samples comprises: at least one verification sample belonging to the same subgroup as the training sample in the ith sample subgroup; and for the ith group of verification samples, acquiring a calibration temperature which enables the calibration error of the AI classification model on the ith group of verification samples to meet the condition, and taking the calibration temperature as the calibration temperature corresponding to the ith sample subgroup.

In some embodiments, the subgroup temperature determination module 760 is configured to obtain an initialized calibration temperature;

obtaining classification results respectively corresponding to all verification samples contained in the ith group of verification samples through the AI classification model; in the kth iteration process, calibrating the classification result corresponding to the ith group of verification samples according to the calibration temperature of the kth iteration to obtain the calibrated classification result corresponding to the ith group of verification samples; wherein k is a positive integer, the initial value of k is 1, and the calibration dimension of the 1 st iteration is the initialized calibration temperature; determining a calibration error corresponding to the ith group of verification samples according to the calibrated classification result corresponding to the ith group of verification samples and the label information corresponding to the ith group of verification samples; adjusting the calibration temperature of the kth iteration with the aim of minimizing the calibration error corresponding to the ith group of verification samples to obtain the calibration temperature of the kth+1th iteration; if the calibration error corresponding to the ith group of verification samples does not meet the condition, k=k+1, and calibrating the classification result corresponding to the ith group of verification samples again from the calibration temperature according to the kth iteration to obtain the calibrated classification result corresponding to the ith group of verification samples; and if the calibration error corresponding to the ith group of verification samples meets the condition, determining the calibration temperature of the (k+1) th iteration as the calibration temperature corresponding to the ith sample subgroup.

In some embodiments, the result calibration module 740 is configured to obtain, through a fully connected layer of the AI classification model, a feature mapping result corresponding to a feature representation of the test sample; and determining the confidence coefficient of the test sample after calibration under the multiple categories according to the ratio of the feature mapping result to the calibration temperature of the test sample, and obtaining the calibrated classification result.

According to the technical scheme provided by the embodiment of the application, the calibration temperature of the test sample is determined through the calibration temperatures respectively corresponding to the n sample subgroups, and the confidence degrees of the test sample obtained by the AI model under a plurality of categories are calibrated according to the calibration temperature of the test sample, so that the calibrated classification result is obtained. Each sample subgroup has a corresponding calibration temperature, and under the condition of subgroup distribution migration, all sample subgroups can obtain good calibration performance, so that the accuracy and reliability of decision calibration are ensured, and the accuracy of an AI classification model is further improved.

Referring to fig. 9, a block diagram of a computer device according to an embodiment of the present application is shown. The computer device is used for implementing the AI model-based classification method provided in the above-described embodiment. Specifically, the present application relates to a method for manufacturing a semiconductor device.

The computer apparatus 900 includes a CPU (Central Processing Unit ) 901, a system Memory 904 including a RAM (Random Access Memory ) 902 and a ROM (Read-Only Memory) 903, and a system bus 905 connecting the system Memory 904 and the central processing unit 901. The computer device 900 also includes a basic I/O (Input/Output) system 906, which helps to transfer information between various devices within the computer, and a mass storage device 907, for storing an operating system 913, application programs 914, and other program modules 915.

The basic input/output system 906 includes a display 908 for displaying information and an input device 909, such as a mouse, keyboard, etc., for user input of information. Wherein the display 908 and the input device 909 are connected to the central processing unit 901 via an input output controller 910 connected to the system bus 905. The basic input/output system 906 can also include an input/output controller 910 for receiving and processing input from a number of other devices, such as a keyboard, mouse, or electronic stylus. Similarly, the input-output controller 910 also provides output to a display screen, a printer, or other type of output device.

The mass storage device 907 is connected to the central processing unit 901 through a mass storage controller (not shown) connected to the system bus 905. The mass storage device 907 and its associated computer-readable media provide non-volatile storage for the computer device 900. That is, the mass storage device 907 may include a computer readable medium (not shown) such as a hard disk or CD-ROM (Compact Disc Read-Only Memory) drive.

The computer readable medium may include computer storage media and communication media without loss of generality. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes RAM, ROM, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory, erasable programmable read-only memory), flash memory or other solid state memory, CD-ROM, DVD (Digital Video Disc, high density digital video disc) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage medium is not limited to the one described above. The system memory 904 and mass storage device 907 described above may be collectively referred to as memory.

According to various embodiments of the application, the computer device 900 may also operate by being connected to a remote computer on a network, such as the Internet. I.e., the computer device 900 may be connected to the network 912 through a network interface unit 911 coupled to the system bus 905, or other types of networks or remote computer systems (not shown) may be coupled using the network interface unit 911.

The memory has stored therein a computer program that is loaded and executed by the processor to implement the AI model-based classification method described above.

In an exemplary embodiment, a computer readable storage medium is also provided, in which a computer program is stored which, when being executed by a processor, implements the above-mentioned AI model-based classification method.

Alternatively, the computer-readable storage medium may include: ROM (Read-Only Memory), RAM (Random-Access Memory), SSD (Solid State Drives, solid State disk), optical disk, or the like. The random access memory may include ReRAM (Resistance Random Access Memory, resistive random access memory) and DRAM (Dynamic Random Access Memory ), among others.

In an exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program stored in a computer readable storage medium. The processor of the computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-described AI model-based classification method.

It should be noted that, the information (including but not limited to user equipment information, user personal information, etc.), data (including but not limited to data for analysis, stored data, presented data, etc.), and signals related to the present application are all authorized by the user or are fully authorized by the parties, and the collection, use, and processing of the related data is required to comply with the relevant laws and regulations and standards of the relevant countries and regions. For example, in the embodiment of the present application, the AI classification model infers a classification result corresponding to a medical image according to the medical image, evaluates a decision result corresponding to the medical image, and may need to acquire the medical image of the user. Wherein the medical images of the user involved are acquired with sufficient authorization.

It should be understood that references herein to "a plurality" are to two or more. "and/or", describes an association relationship of an association object, and indicates that there may be three relationships, for example, a and/or B, and may indicate: a exists alone, A and B exist together, and B exists alone. The character "/" generally indicates that the context-dependent object is an "or" relationship.

The foregoing description of the exemplary embodiments of the application is not intended to limit the application to the particular embodiments disclosed, but on the contrary, the intention is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the application.

Claims

1. A classification method based on an artificial intelligence AI model, the method comprising:

2. The method of claim 1, wherein determining the weight value for each of the n sub-groups of samples based on the characteristic representation of the test sample as sub-group characteristics for each of the n sub-groups of samples, comprises:

respectively calculating distance values between characteristic representations of the test samples and subgroup characteristics corresponding to each sample subgroup to obtain distance values corresponding to the n sample subgroups; the distance value corresponding to the ith sample subgroup is used for representing the proximity between the characteristic representation of the test sample and the subgroup characteristic corresponding to the ith sample subgroup;

And determining the weight values corresponding to the n sample subgroups according to the distance values corresponding to the n sample subgroups respectively.

3. The method of claim 1, wherein determining the calibration temperature of the test sample based on the weight values respectively corresponding to the n sub-groups of samples and the calibration temperatures respectively corresponding to the n sub-groups of samples comprises:

and carrying out weighted summation on the calibration temperatures corresponding to the n sample subgroups respectively according to the weight values corresponding to the n sample subgroups respectively to obtain the calibration temperature of the test sample.

4. The method according to claim 1, wherein the method further comprises:

dividing training samples contained in a training sample set into n sample subgroups;

respectively extracting characteristic representations of the training samples through the AI classification model;

and for the ith sample subgroup in the n sample subgroups, averaging the feature representations of the training samples contained in the ith sample subgroup to obtain subgroup features corresponding to the ith sample subgroup.

5. The method according to claim 1, wherein the method further comprises:

Performing subgroup division on verification samples contained in the verification sample set to obtain n groups of verification samples; wherein an ith set of validation samples of the n sets of validation samples comprises: at least one verification sample belonging to the same subgroup as the training sample in the ith sample subgroup;

and for the ith group of verification samples, acquiring a calibration temperature which enables the calibration error of the AI classification model on the ith group of verification samples to meet the condition, and taking the calibration temperature as the calibration temperature corresponding to the ith sample subgroup.

6. The method of claim 5, wherein the obtaining a calibration temperature that causes a calibration error of the AI classification model on the ith set of verification samples to satisfy a condition as a calibration temperature for the ith sub-group of samples comprises:

acquiring an initialized calibration temperature;

obtaining classification results respectively corresponding to all verification samples contained in the ith group of verification samples through the AI classification model;

in the kth iteration process, calibrating the classification result corresponding to the ith group of verification samples according to the calibration temperature of the kth iteration to obtain the calibrated classification result corresponding to the ith group of verification samples; wherein k is a positive integer, the initial value of k is 1, and the calibration dimension of the 1 st iteration is the initialized calibration temperature;

Determining a calibration error corresponding to the ith group of verification samples according to the calibrated classification result corresponding to the ith group of verification samples and the label information corresponding to the ith group of verification samples;

adjusting the calibration temperature of the kth iteration with the aim of minimizing the calibration error corresponding to the ith group of verification samples to obtain the calibration temperature of the kth+1th iteration;

if the calibration error corresponding to the ith group of verification samples does not meet the condition, k=k+1, and calibrating the classification result corresponding to the ith group of verification samples again from the calibration temperature according to the kth iteration to obtain the calibrated classification result corresponding to the ith group of verification samples;

and if the calibration error corresponding to the ith group of verification samples meets the condition, determining the calibration temperature of the (k+1) th iteration as the calibration temperature corresponding to the ith sample subgroup.

7. The method according to any one of claims 1 to 6, wherein calibrating the confidence level of the test sample under a plurality of categories obtained by the AI classification model according to the calibration temperature of the test sample, to obtain a calibrated classification result, includes:

Acquiring a feature mapping result corresponding to the feature representation of the test sample through the full connection layer of the AI classification model;

and determining the confidence coefficient of the test sample after calibration under the multiple categories according to the ratio of the feature mapping result to the calibration temperature of the test sample, and obtaining the calibrated classification result.

8. An AI model-based classification apparatus, the apparatus comprising:

the characteristic extraction module is used for extracting characteristic representations of test samples in text, voice or image forms through the AI classification model;

9. A computer device comprising a processor and a memory, wherein the memory has stored therein a computer program that is loaded and executed by the processor to implement the AI model-based classification method of any of claims 1-7.

10. A computer-readable storage medium, in which a computer program is stored, which is loaded and executed by a processor to implement the AI model-based classification method of any of claims 1-7.