CN117557840A

CN117557840A - A method for grading fundus lesions based on small sample learning

Info

Publication number: CN117557840A
Application number: CN202311491052.9A
Authority: CN
Inventors: 徐晓; 李甦雁; 吴亮; 余颖; 杨旭; 牛亮; 曹咏琦; 李蕴龙; 牛强
Original assignee: FIRST PEOPLE'S HOSPITAL OF XUZHOU; China University of Mining and Technology CUMT
Current assignee: FIRST PEOPLE'S HOSPITAL OF XUZHOU; China University of Mining and Technology CUMT
Priority date: 2023-11-10
Filing date: 2023-11-10
Publication date: 2024-02-13
Anticipated expiration: 2043-11-10
Also published as: CN117557840B

Abstract

The invention discloses a fundus disease grading method based on small sample learning, which comprises the steps of firstly, collecting and preprocessing retina fundus color photographs to obtain the existing fundus disease data set; respectively learning the intra-class characteristics and inter-class characteristics of each level of fundus lesions by the image data through a contrast network and a meta-network which are subjected to pre-training and meta-training; and finally, carrying out similarity scoring on the unlabeled image and each level fundus lesion prototype so as to carry out grading prediction of fundus lesions. According to the invention, fundus lesions can be classified better by self-learning fundus lesions and fundus illumination with a small number of samples through a dual-network structure comprising a comparison network and a meta-network after meta-training, and the fundus lesions can be classified on the basis of effectively reducing noise influence in both aspects of a feature space and a label space, so that the accuracy of fundus lesions prediction is improved.

Description

Fundus lesion grading method based on small sample learning

Technical Field

The invention relates to a fundus lesion grading method based on image data, in particular to a fundus lesion grading method based on small sample learning, and belongs to the technical field of medical image processing and computer vision.

Background

In recent years, deep learning techniques have been widely used in various fields such as computer vision and the like and have achieved remarkable effects. The high accuracy of this is highly dependent on large scale marking data, which is not always available in real life for large amounts of tagged data, e.g. in the medical field. Taking fundus disease prediction as an example, various diseases such as diabetic retinopathy, glaucoma, transformation of contralateral eyes to neovascular AMD within one year, cardiovascular diseases (ischemic stroke, myocardial infarction, heart failure) and neurodegenerative diseases (Parkinson's disease) and the like can be diagnosed and predicted by fundus illumination. Early prediction results may allow timely intervention, especially for patients with chronic diseases, such as diabetics, and early disease prediction may help prevent serious complications. However, the retinal fundus color photograph is difficult to obtain in various technologies and physiology, so that the problems of pupil size, patient cooperation, corneal opacity, refractive error and the like can lead to incapability of collecting pictures, and the quality of used equipment and lenses is important for obtaining high-quality fundus color photographs. Low quality equipment or lenses may cause distortion and distortion of images, while high quality fundus photographing equipment is generally expensive and has limited medical resources, so that tagged fundus lesion data used as a training sample is relatively less, a traditional deep learning method is difficult to use, and further, the acquired fundus color photographs are difficult to judge and screen diseases, and a great deal of manpower and time are required.

Small sample learning, which may imitate humans using few examples in tasks to identify new classes, is of increasing interest due to the high cost and effort of collecting large amounts of data. The purpose of the small sample learning is to perform quick learning by only a small number of marked data samples, so that the small sample learning has good generalization performance on new tasks. However, most of the existing small sample learning methods are based on the assumption that label information is completely clean and complete, and the robustness of a model facing a noise label is not considered, in fact, the noise label is ubiquitous due to limited knowledge and unintentional damage in medical images, so that the problem of low accuracy of a prediction result exists in fundus lesion prediction classification by utilizing the existing small sample learning methods. Taking grading of a sugar network (short for diabetic retinopathy) as an example, on one hand, grading standards are easy to be confused when grading samples with the same judging standard and different grades, for example, in light NPDR and medium NPDR, the judgment can be made by virtue of microangioma, but the difference is that the degree is light and heavy, so that the fundus color photograph angle is easy to identify errors for samples like transition from light NPDR to medium NPDR; on the other hand, because the fundus color photograph of the patient possibly contains unfamiliar or rare disease features of the expert, and reasons such as photographing angles, data transmission, damage and the like of the fundus color photograph exist, the expert often generates deviation in grading labeling, and further the noise data of labeling samples are generated, and the existence of noise often causes large model deviation, so that a prediction result is seriously influenced.

Disclosure of Invention

Aiming at the problems in the prior art, the invention provides a fundus lesion grading method based on small sample learning, which can realize fundus lesion grading on the basis of effectively reducing noise influence in both a characteristic space and a label space, thereby improving the accuracy of fundus lesion prediction.

In order to achieve the purpose, the fundus lesion grading method based on small sample learning specifically comprises the following steps:

step1, collecting retinal fundus color photographs from a data source, deriving pictures after the data acquisition is completed, and marking all the pictures by a professional doctor, and grading according to international clinical grading standards of fundus lesions;

step2, selecting data set after processing the eye bottom color photograph to improve image quality, reduce noise and normalize image, and selecting K samples for each level as support setQuery set->Wherein y is _i ∈C _novel K is the number of samples extracted for each fundus lesion level, M is the number of samples of query set Q;

at the same time, an auxiliary data set with rich samples and accurate labels is definedWherein y is _i ∈C _base Requirement C _base ∩C _novel Let phi, also divide the auxiliary data set into support sets S _b And query set Q _b The meta-training device is used for performing meta-training;

step3, constructing a comparison network for generating intra-class weights, and learning intra-class characteristics of each level of fundus lesions after pre-training and meta-training;

step4, constructing a meta-network for generating inter-class weights, and learning inter-class characteristics of each level of fundus lesions after pre-training and meta-training;

step5, correcting the eye fundus lesion prototype by using the sample weights extracted by the comparison network and the meta-network;

step6, carrying out similarity scoring on the unlabeled images and each grade fundus lesion prototype so as to carry out grading prediction of fundus lesions.

Further, step3 specifically comprises the following steps:

step3-1, pre-training contrast network g _ξ Mapping the sample vector into a certain feature space through a feature extraction network;

step3-2, meta-training stage for a more widely sourced medical dataset D _b K samples under each category and K samples classified by each fundus lesion in the query stage are calculated according to cosine similarity, and the similarity between every two samples is calculated according to the following calculation formula:

cor(x _a ,x _b )＝cos(g _ξ (x _a ),g _ξ (x _b ))

wherein: g _ξ Representing a pre-trained contrast network feature extractor, x _a ,x _b Respectively representing two support samples;

obtaining a K x K correlation matrix Z for a rank n _n ∈R ^K×K Each correlation matrix Z _n The method comprises the steps of including relevant information between a certain support sample and the rest K-1 samples of the same level, wherein the information is scattered on other K-1 relevant characteristics at the same time;

step3-3, correlation matrix Z _n Direct input transducer layer:

wherein: phi (phi) _T Z is a transducer layer parameter _n Represents a correlation matrix, o _n Representing an output result;

then using K sample softmax function to calculate the weight vector V in the class corresponding to each sample _n ：

V _n ＝ρ(o _n )

A vector is generated for each sample that characterizes the weights within the class.

Further, step3-1 is specifically as follows:

step3-1-1, feature extractor pre-training: given feature extractor f _θ (. Cndot.) prototype for each level k is calculated as

Wherein:the number of the support samples with the fundus lesion level k; x is x _t Representing a support sample; y is _t Representing sample x _t A corresponding fundus lesion level label;

new sample x for a given set of queries _q The classifier outputs a normalized classification score for each class k

Wherein: sim (sim) _w (. Cndot.) is a similarity function; f (f) _θ (. Cndot.) is a feature extractor, x _q To query the sample, c _k A prototype of class k;

sim _w (. Cndot.) the following calculation was used

sim _w (A,B)＝λcos(F _w (A),F _w (B))

Wherein: f (F) _w (.) is a w-parameterized single-layer neural network; lambda is the inverse temperature parameter;

the θ and ω are updated by the following loss function

Wherein: x is x _q ,y _q All derived from the query sample; p is p _θ,ω Normalized scores for the corresponding categories; e represents a mathematical expectation;

step3-1-2, contrast web pretraining:

contrasting network g _ξ Training with condition loss and contrast loss, giving input fundus color photograph x, generating two images x 'and x' processed by different data enhancement methods, sequentially placing the two enhanced images into contrast feature extraction network g _ξ And a projected multi-layer perceptron head sigma for comparing the network g _ξ The extracted original features are embedded for further projection and transformation, and finally, the mapped result after sigma is put into a predictive multi-layer perception machine head delta to obtain the sine and cosine similarity of two embedded vectors:

wherein: I.I ₂ Represents the l2 norm; stop-gradient is stop gradient operation;

at the same time, the condition loss is utilized to guide the learning of the comparison network, and the comparison network is composed of a feature extractor f _θ (. Cndot.) guide learning:

combining the comparison loss with the conditional loss to obtain an objective function of the comparison network optimization, wherein the objective function is as follows:

wherein: gamma is a normal number, balancing the importance of contrast loss and conditional loss.

Further, step4 specifically comprises the following steps:

step4-1, initializing a meta learning networkFor extracting support set features;

step4-2, firstly, for all support samples, carrying out K-Means clustering according to a class prototype of the support sample as an initial cluster center until convergence;

step4-3, calculating cosine similarity between the final clustering center and all samples to obtain a similarity matrix;

step4-4, obtaining a certain support sample x by using a softmax function _t The inter-class weights for a certain fundus lesion classification are:

wherein:representing sample x _t Similarity score for fundus lesion level n; τ is a superparameter to prevent gradient disappearance; k represents the number of samples corresponding to each category.

Further, step5 specifically comprises the following steps:

step5-1, prototype correction in comparison network is:

wherein: α+β=1;is an inter-class weight; />Is an intra-class weight; g _ξ Representing a comparison network feature extractor; x is x _t Represents the t-th sample; />A kth sample representing a class n;

step5-2, prototype modification in the meta-network is:

wherein: α+β=1;is an inter-class weight; />Is an intra-class weight; />Representing a meta-network feature extractor; x is x _t Represents sample t, ++>Represents the kth sample of class n.

Further, step6 specifically comprises the following steps:

step6-1, define the EC similarity score as follows:

wherein: x represents a query sample; c _n A sample center point representing the fundus lesion level; g _ξ Representing a comparison network feature extractor;representing a meta-network feature extractor; p is p _n Representing the prototype corrected by the comparison network; p's' _n Representing the prototype after the correction of the meta-network;

step6-2, calculating EC similarity scores corresponding to various levels of fundus lesions according to a certain query sample x, and obtaining probabilities corresponding to the fundus lesion levels by using a softmax function:

wherein: s is S _EC Representing EC similarity scores;representing a query sample; c _n A sample center point representing the fundus lesion level; t is a superparameter.

Compared with the prior art, the fundus lesion grading method based on small sample learning has the following advantages:

1. in the fundus disease prediction process, the acquisition difficulty of a data set is high, so that the number of samples trained by an artificial neural network is limited and is insufficient to train a traditional neural network.

2. Because of the problems of high difficulty in obtaining high-quality retina fundus color photographs, unknown diseases affecting the identification of fundus lesions, and the like, the grading labeling of the fundus lesions by an expert may deviate. The invention eliminates unreasonably marked noise images as far as possible by constructing a double-network structure and tries to classify the fundus lesions correctly. The contrast network and the meta network respectively correct the fundus lesion prototype of each level through the intra-class weight and the inter-class weight, so that the characteristic and the label representation capability of the model are improved. Metric level calibration can mitigate the effects of two network mispredictions, improving the stability and robustness of the model.

Drawings

FIG. 1 is a schematic overall flow diagram of the present invention;

FIG. 2 is a block diagram of the data preprocessing of the present invention;

FIG. 3 is an overall framework of a dual network architecture in accordance with the present invention;

FIG. 4 is a schematic diagram of the generation of intra-class weights and inter-class weights in a dual network architecture of the present invention;

FIG. 5 is a schematic diagram of an original modification strategy in a dual network architecture of the present invention;

fig. 6 is a schematic diagram of the comparative network pre-training phase of the present invention.

Detailed Description

The invention will be further described with reference to the accompanying drawings, taking the grading of the lesions of the sugar network as an example.

As shown in fig. 1, firstly, collecting and preprocessing retina fundus color photographs, on one hand, enabling images to adapt to network input, and on the other hand, facilitating subsequent training processes to reduce calculated amount and speed up training, and obtaining an existing sugar network lesion data set; respectively learning the intra-class characteristics and inter-class characteristics of each level of sugar network lesions by the image data through a pre-trained and meta-trained comparison network and a meta-network, and reducing the influence of noise fundus color illumination with labeling deviation; and finally, carrying out similarity scoring on the unlabeled images and each level of sugar net prototype so as to carry out grading prediction of the sugar net. The method comprises the following steps:

step1, data acquisition and labeling

Retinal fundus color illumination is collected from data sources such as diabetics and healthy subjects. The data collection may be performed using sensors, medical devices, or database queries, among other means. The pictures are derived after the data acquisition is completed, and marked by a professional doctor, and the pictures are classified into no obvious retinopathy (grade I), mild NPDR (grade II), moderate NPDR (grade III), severe NPDR (grade IV) and PDR (grade V) according to the new international clinical grading standard of diabetic retinopathy. The specific grading criteria and mydriatic fundus examination are shown in Table 1 below.

Table 1 diabetic retinopathy grading criteria

Step2, data preprocessing

The method is characterized in that data preprocessing is performed before the analysis of the eye fundus color illumination, so that the image quality is improved, noise is reduced, and the image is standardized, and the subsequent analysis is facilitated. As shown in fig. 2, the data preprocessing employs the following steps:

step2-1, image sharpening: by applying the image sharpening filter, edges and details of the image can be enhanced, improving the sharpness of the image. Common sharpening filters include Sobel, laplacian and Gao Sirui sharpening.

Step2-2, contrast enhancement: increasing the contrast of the image may make the lesions more visible. The contrast enhancement method includes histogram equalization and contrast stretching.

Step2-3, denoising: fundus illumination may be subject to various noise, such as light noise, artifacts, and the like. Denoising methods include median filtering, gaussian filtering, and wavelet denoising.

Step2-4, color normalization: the color and brightness of fundus images may be different depending on different photographing conditions, and thus color normalization is required to keep the color and brightness uniform between different images.

Step2-5, removing fundus reflection: the optic disc (central portion of the fundus) in fundus illumination typically introduces large changes in luminance, sometimes requiring removal or alleviation of the effects of this area in order to better analyze other portions of the retina.

Step2-6, image cropping: the image may be cropped to preserve only the region of interest (ROI) as needed for a particular task, thereby reducing the complexity of the process.

Step2-7, standardization of image scale: the images are adjusted to the same size for training and analysis of the deep learning model.

Step2-8, selecting a dataset: k samples are selected for each grade of the sugar net grading five-grade standardTo support a collection, i.eQuery set->Wherein y is _i ∈C _novel K is the number of samples extracted for each grade of sugar net lesion, and M is the number of samples of query set Q. Each FSL problem with the support set estimating sample categories in the query set can be seen as a task. At the same time, an auxiliary dataset is defined which has a rich sample and is labeled accurately +.>Wherein y is _i ∈C _base The data set may be a data set derived from other fundus medical tasks, requirement C _base ∩C _novel =Φ, the auxiliary dataset is also divided into a support set and a query set: s is S _b And Q _b They are used for meta-training. This strategy can be seen as being in base class C _base Training the model on the data of (1) to make the model generalize well to C _novel 。

Step3, constructing a comparison network

In order to cope with the interference of noise information on model training effect under small sample learning, a dual-network architecture is constructed, which consists of a comparison network and a meta-network, wherein the comparison network is used for generating weights in classes, and the meta-network is used for generating weights between classes. A specific architecture is shown in fig. 3.

Step3-1, pre-training contrast network g _ξ (see Step7 for details) the sample vector is mapped into a feature space through the feature extraction network.

Step3-2, meta-training stage for a more widely sourced medical dataset D _b K samples under each category and K samples classified by each sugar net in the query stage are calculated according to cosine similarity, and the similarity between every two samples is calculated according to the following calculation formula:

cor(x _a ,x _b )＝cos(g _ξ (x _a ),g _ξ (x _b )).

thus, a K X K correlation matrix Z can be obtained for the rank n _n ∈R ^K×K For example Z ₁ The overall correlation between the sample with the sugar network classified as class I and the class I sugar network is disclosed. Modeling can be performed with potential correlation between correlated samples of the same level. Each correlation matrix Z _n The method comprises the steps of including the related information between a certain supporting sample and the rest K-1 samples of the same level, and dispersing the information on other K-1 related characteristics. The nature of the interconnect makes it more reasonable to fully consider the context of the relevant features of the same class when generating the intra-class weights.

Step3-3, the self-attention mechanism in the transducer model utilizes support samples to assign weights based on the similarity between them. Specifically, a correlation matrix among samples is obtained after the Step3-2, and a correlation matrix Z is obtained _n Direct input transducer layer (without position coding):

V _n ＝ρ(o _n )

After steps 3-1, step3-2, step3-3, a vector representing the weights within the class can be generated for each sample. The overall process of specifically generating such internal weights is shown in fig. 4.

Step4, constructing a meta-network

Step4-1, initializing a meta learning networkFor extracting support set features.

Step4-2, firstly, for all the support samples, carrying out K-Means clustering by taking a class prototype of the support sample as an initial cluster center according to the fact that the cluster is five, until convergence. Class prototypes are representative representations generated based on support samples, and the center or average value of each class may be taken as the class prototype in the present invention.

Step4-3, calculating the cosine similarity between the final clustering center and all samples, and consistent with Step 4-2. Five clustering centers can be obtained through Step4-1, cosine similarity between each clustering center and all samples is calculated, and S epsilon R can be finally obtained ^25×K Is a similar matrix of (a) for the image.

Step4-4, obtaining a certain support sample x by using a softmax function _t (t=1, 2,3., 25K), which is the inter-class weight for a certain sugar net classification:

wherein:representing sample x _t Similarity score for sugar network class n; τ is a superparameter to prevent gradient disappearance; k represents the number of samples corresponding to each category.

Step5, prototype modification

A frame diagram of the prototype modification strategy is shown in fig. 5.

Step5-1, prototype correction in comparison network is:

wherein: α+β=1;is composed ofstep3-3, obtaining an inter-class weight; />Is an intra-class weight obtained from step 4-4; g _ξ Representing a comparison network feature extractor; x is x _t Represents the t-th sample; />Represents the kth sample of class n.

Step5-2, prototype modification in the meta-network is:

wherein: α+β=1;the inter-class weight is obtained from step 3-3; />Is an intra-class weight obtained from step 4-4; />Representing a meta-network feature extractor; x is x _t Represents sample t, ++>Represents the kth sample of class n.

Step6, similarity score

To mitigate the effects of two network mispredictions, we introduced metric intelligent calibration on both networks, using a method called EC similarity to help calculate the predicted scores for each level of sugar network. As shown in fig. 6, the calculation of a specific EC similarity score includes the following steps:

step6-1, define the EC similarity score as follows:

wherein: x represents a query sample; c _n A sample center point representing the sugar network level; g _ξ Representing a comparison network feature extractor;representing a meta-network feature extractor; p is p _n Representing the prototype corrected by the comparison network; p's' _n Representing the prototype after the meta-network modification.

That is, for each level of the sugar network disorder, the incoming query sample x (i.e., retinal fundus color illumination) passes through the comparison network g _ξ And meta-networkAfter mapping of (a), the corresponding EC score can be calculated.

Step6-2, calculating EC similarity scores corresponding to the I/II/III/IV/V grade sugar net lesions according to a certain query sample x, and obtaining probabilities corresponding to the sugar net lesion grades by using a softmax function:

Step7, model Pre-training

Step7-1, feature extractor pre-training: a prototype for each class may be computed by a feature extractor. Given feature extractor f _θ (. Cndot.) prototypes for each level k can be calculated as

Wherein:the number of the support samples with the sugar net level of k; f (f) _θ Implemented by Conv-4-64 backbone network, x _t Representing a support sample; y is _t Representing sample x _t Corresponding labels, i.e. samples x _t Is a sugar network grade of (2);

Wherein: sim (sim) _w (. Cndot.) is a similarity function; f (f) _θ (. Cndot.) is a feature extractor, x _q To query the sample, c _k Is a prototype of class k.

sim _w (. Cndot.) the following calculation was used

sim _w (A,B)＝λcos(F _w (A),F _w (B))

Wherein: f (F) _w (.) is a w-parameterized single-layer neural network, the output dimension is 2048, and λ is the inverse temperature parameter.

The θ and ω are updated by the following loss function

Wherein: x is x _q ,y _q All derived from the query sample, p _θ,ω For the normalized score of the corresponding category, the calculation method is as above; e represents a mathematical expectation.

Step7-2, comparative web pre-training:

contrasting network g _ξ Training can be performed with conditional and comparative losses. Given an input fundus color photograph x, two images x 'and x' processed by different data enhancement methods are generated, and the two enhanced images are sequentially put into a contrast feature extraction network g _ξ And a projected multi-layer perceptron header sigma for extracting the extracted features from the feature extractor (i.e. the contrast feature extraction network g _ξ ) The extracted original features are embedded for further projection and transformation. Finally, the result mapped after sigma is put into a predictive multi-layer perception machine head delta, and then the sine and cosine similarity of two embedded vectors can be obtained:

wherein: I.I ₂ Represents the l2 norm; stop-gradient is a stop gradient operation that is commonly used in the construction of certain loss functions, where certain parameters should not be updated according to the gradient, but should remain unchanged.

At the same time, the condition loss is utilized to guide the learning of the comparison network, and the comparison network is composed of a feature extractor f _θ (. Cndot.) (see Step 7-1) guide learning:

f _θ Realized by Conv-4-64 backbone network, the projection multi-layer perception machine head sigma is formed by a single-layer neural network F _w (-) single-layer neural network and one multi-layer neural network (each layer of neural network has {1600,2048,2048} units and is subjected to batch normalization). The predictive multi-layer perceptron head delta is parameterized by a three-layer neural network having 512 hidden units and batch normalized at the hidden layers, all using the ReLU function as the activation function.

Step8, model element training

The training set used in the meta-training phase is derived from the broader medical data set D _b It requires no sample of the used set of sugar network lesions, i.e. C _base ∩C _novel ＝φ，D _b The number of categories included is N. Step1 to Step7 show how the model recognizes the grading of the sugar network (i.e. in case of n=5), for a medical dataset D of more extensive origin _b The steps are consistent.

Step8-1, define meta-loss:

wherein: m is the total number of samples of the query set,

in dataset D _b The artificial noise is defined by defining data set D _b The noise loss in the above class is as follows:

wherein whenFor artificially induced noise +.>Otherwise, 0.

For all support set samples, constructing a similarity matrix M, calculating the similarity between any two samples, and defining the inter-class loss as:

wherein,

l(x ⁽ⁱ⁾ ) Represents x ⁽ⁱ⁾ Is a true value for the tag.

The final loss function is defined as:

L _total ＝L _me +ηL _ra +γL _er

wherein: η and γ are normal numbers representing the importance of the different losses.

Step9, diabetes stratification

The existing marked sugar net graded picture (support set S _n Derived from C _novel ) Put into the network, the new sugar network dataset will learn itself for the network model that has been trained in step 8.

Finally, when grading prediction is carried out on single fundus color photographs, the EC similarity is used for measuring the score of each grade, and the grade with the highest score is obtained as a final prediction label:

the probability of each level of sugar net symptom can be calculated using the following formula:

the label distribution of the symptoms of the sugar network of each grade is calculated.

The method can be implemented by using a hardware platform such as a computer, a server, a mobile device and the like, wherein data processing and model training are involved. The method may also incorporate real-time monitoring equipment and a patient database to continuously track and update the grading results. Contrast network in denoising networkAnd element networkCollaterals g _ξ All can be implemented by ConvNet model (C64E) backbone. In C64E, each block consists of 64-channel 3×3 convolution, batch normalization, reLU nonlinearity, and 2×2 max pooling. The feature embedding dimension is set to 1600.

The fundus lesion grading method based on small sample learning removes the data with marking deviation in fundus color photographs through a double-network structure formed by a comparison network and a meta-network, and performs fundus lesion grading prediction on fundus color photographs in query concentration on the basis. The dual network architecture consisting of the comparison network and the meta network is calibrated in two ways to eliminate data noise: example level calibration and metric level calibration. In the aspect of example-level calibration, the prototype is corrected by utilizing the sample weights extracted by two networks, so that the relation between sugar net samples of each level and the whole sugar net samples of different levels can be better captured, the distinguishing capability of the model on the samples of different levels is improved, the corrected prototype can more accurately represent the characteristic and label information of each sugar net level, the performance of the model in the classification learning task of the sugar net with fewer samples is improved, and the characteristic representation capability and label representation capability of the model are improved. In terms of metric level calibration, ensemble with Consistency (EC) principle is introduced, similarity between two examples is calculated by fusing similarity evaluation results in two different networks, EC similarity can adjust confidence of similarity prediction according to consistency of the similarity in the two networks and scale similarity prediction scores, similarity between two sugar net samples can be evaluated more accurately by using EC similarity, specifically, the similarity score of each sugar net level can be scaled implicitly, the prediction scores can be calibrated adaptively by calculating consistency of the similarity scores of the two network predictions, and the metric level calibration can alleviate influence of two network mispredictions and improve stability and robustness of a model.

The artificial neural network comprises a pre-training module, adopts a self-supervision learning method, utilizes supervision information in marked data, shapes and improves self-supervision learning feature manifolds under the condition of no auxiliary unmarked data, reduces characterization deviation, and mines more effective semantic information. When the comparison network is pre-trained, the feature extractor is firstly learned on the labeling data through the original supervised learning method, and the prototype of each category is calculated, namely, the prototype of each sugar network level is calculated. Next, a conditional self-monitoring model is trained using the self-monitoring module and the condition module. The self-supervision module generates two different enhancement views by using a random enhancement method, and calculates the similarity loss between the embedded vectors, namely the similarity loss between the sugar net lesion images of different visual angles of the same patient. The condition module uses the features learned in the pre-training stage as priori knowledge, namely uses the prototype representation of each learned sugar net level as guidance, optimizes the feature manifold learned by the self-supervision module, and enables the model to extract more semantic information in combination with the multiparty view angle in the comparison network so as to obtain better representation.

Claims

1. A fundus lesion grading method based on small sample learning, which is characterized by including the following steps:

Step 1. Collect retinal fundus color photos from the data source. After the data collection is completed, export the pictures and submit them to professional doctors to label all pictures and classify them according to the international clinical grading standards for fundus lesions;

Step 2. After processing the fundus color photos to improve image quality, reduce noise, and standardize the images, select the data set, and select K samples for each level as the support set. Queryset/> Among them, y _i ∈C _novel , K is the number of samples extracted for each fundus lesion level, and M is the number of samples in the query set Q;

At the same time, define an auxiliary data set with rich samples and accurate annotation. Among them, y _i ∈C _base requires C _base ∩C _novel =φ. The auxiliary data set is also divided into a support set S _b and a query set Q _b for meta-training;

Step 3: Construct a comparison network for generating intra-class weights, and after pre-training and meta-training, learn the intra-class features of fundus lesions at each level;

Step 4: Construct a meta-network for generating inter-class weights, and after pre-training and meta-training, learn the inter-class features of fundus lesions at each level;

Step 5: Use the sample weights extracted by the comparison network and meta-network to correct the fundus lesion prototype;

Step 6: Score the similarity between the unlabeled images and the prototypes of fundus lesions at each level to predict the grade of fundus lesions.

2. The fundus lesion grading method based on small sample learning according to claim 1, characterized in that the specific process of Step 3 is as follows:

Step3-1, pre-train the comparison network _gξ , and map the sample vector to a certain feature space through the feature extraction network;

Step 3-2. In the meta-training stage, for the K samples corresponding to each category of the medical data set D _b from a wider range of sources and the query stage for the K samples of each fundus lesion classification, each pair is calculated according to the cosine similarity. The similarity between samples is calculated as follows:

cor(x _a ,x _b )=cos(g _ξ (x _a ),g _ξ (x _b ))

In the formula: g _ξ represents the pre-trained comparison network feature extractor, x _a and x _b respectively represent two support samples;

Obtain K×K correlation matrix Z _n ∈R ^K×K for level n. Each correlation matrix Z _n contains relevant information between a certain support sample and the remaining K-1 samples of the same level. This information is dispersed at the same time. On other K-1 related characteristics;

Step3-3, directly input the correlation matrix Z _n into the Transformer layer:

In the formula: φ _T is the Transformer layer parameter, Z _n represents the correlation matrix, and _on represents the output result;

Then use the K sample softmax function to calculate the intra-class weight vector V _n corresponding to each sample:

V _n =ρ( _on )

For each sample, a vector representing the within-class weights is generated.

3. The fundus lesion grading method based on small sample learning according to claim 2, characterized in that the specific process of Step3-1 is as follows:

Step3-1-1, feature extractor pre-training: given the feature extractor f _θ (·), the prototype of each level k is calculated as

In the formula: is the number of support samples with fundus lesion level k; x _t represents the support sample; y _t represents the fundus lesion level label corresponding to sample x _t ;

Given a new sample x _q of the query set, the classifier outputs a normalized classification score for each class k

In the formula: sim _w (·) is the similarity function; f _θ (·) is the feature extractor, x _q is the query sample, c _k is the support prototype of level k;

sim _w (·) uses the following calculation method

sim _w (A, B) = λ cos (F _w (A), F _w (B))

In the formula: F _w (.) is a single-layer neural network parameterized by w; λ is the inverse temperature parameter;

Then update θ and ω through the following loss function

In the formula: x _q , y _q are all derived from query samples; p _{θ and ω} are the normalized scores of the corresponding categories; E represents mathematical expectation;

Step3-1-2, compare network pre-training:

The contrast network g _ξ is trained with conditional loss and contrast loss. Given the input fundus color photo x, two images x' and x" processed by different data enhancement methods are generated. These two enhanced images are put into comparison successively. The feature extraction network g _ξ and a projected multi-layer perceptron head σ, σ is used to further project and transform the original feature embedding extracted from the comparison network g _ξ , and finally the mapped result after σ is put into a prediction The multi-layer perceptron head δ obtains the negative cosine similarity of the two embedding vectors:

In the formula: ||·|| ₂ represents the l2 norm; stop-gradient is the stop gradient operation;

At the same time, conditional loss is used to guide the learning of the contrast network, which is guided by the feature extractor f _θ (·):

Combining contrastive loss with conditional loss, the objective function of contrastive network optimization is obtained:

In the formula: γ is a positive constant, weighing the importance of contrast loss and conditional loss.

4. The fundus lesion grading method based on small sample learning according to claim 1, characterized in that the specific process of Step 4 is as follows:

Step4-1, initialize meta-learning network Used to extract support set features;

Step 4-2: First, for all support samples, perform K-Means clustering according to the class prototype of the support sample as the initial cluster center until convergence;

Step 4-3: Calculate the final cluster center and calculate cosine similarity with all samples to obtain a similarity matrix;

Step4-4, use the softmax function to obtain the inter-class weight of a certain support sample x _t for a certain fundus lesion classification:

In the formula: Represents the similarity score of sample x _t to fundus lesion level n; τ is a hyperparameter used to prevent gradient disappearance; K represents the number of samples corresponding to each category.

5. The fundus lesion grading method based on small sample learning according to claim 1, characterized in that the specific process of Step 5 is as follows:

Step5-1, compare the prototype in the network and correct it to:

In the formula: α+β=1; Is the weight between classes;/> is the intra-class weight; _gξ represents the comparison network feature extractor; x _t represents the t-th sample;/> Represents the k-th sample of category n;

Step5-2, the prototype in the meta network is revised to:

In the formula: α+β=1; Is the weight between classes;/> Is the weight within the class;/> represents the meta-network feature extractor; x _t represents the t-th sample,/> Represents the k-th sample of category n.

6. The fundus lesion grading method based on small sample learning according to claim 1, characterized in that the specific process of Step 6 is as follows:

Step6-1, define the EC similarity score as follows:

In the formula: x represents the query sample; c _n represents the sample center point of the fundus lesion level; g _ξ represents the contrast network feature extractor; represents the meta-network feature extractor; p _n represents the modified prototype of the comparison network; p' _n represents the modified prototype of the meta-network;

Step6-2: For a certain query sample x, after calculating the EC similarity score corresponding to each level of fundus lesions, use the softmax function to obtain the probability of the corresponding fundus lesion level:

In the formula: S _EC represents the EC similarity score; represents the query sample; c _n represents the sample center point of the fundus lesion level; T is the hyperparameter.