CN111832581A - Lung feature recognition method and device, computer equipment and storage medium - Google Patents
Lung feature recognition method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN111832581A CN111832581A CN202010991495.4A CN202010991495A CN111832581A CN 111832581 A CN111832581 A CN 111832581A CN 202010991495 A CN202010991495 A CN 202010991495A CN 111832581 A CN111832581 A CN 111832581A
- Authority
- CN
- China
- Prior art keywords
- lung
- text
- image
- feature
- recognition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/56—Extraction of image or video features relating to colour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
- G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/259—Fusion by voting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30061—Lung
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/03—Recognition of patterns in medical or anatomical images
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- General Physics & Mathematics (AREA)
- Medical Informatics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Public Health (AREA)
- General Engineering & Computer Science (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- General Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Pathology (AREA)
- Databases & Information Systems (AREA)
- Epidemiology (AREA)
- Multimedia (AREA)
- Primary Health Care (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Radiology & Medical Imaging (AREA)
- Quality & Reliability (AREA)
- Apparatus For Radiation Diagnosis (AREA)
- Image Analysis (AREA)
- Measuring And Recording Apparatus For Diagnosis (AREA)
Abstract
The invention relates to the field of artificial intelligence, and discloses a lung feature identification method, a device, computer equipment and a storage medium, wherein the method comprises the following steps: acquiring data to be identified comprising an image of a lung to be identified and text description of the lung to be identified; extracting lung image features through a lung image recognition model to generate lung image feature vectors and image recognition results, and extracting lung text features through a lung text recognition model to generate lung text feature vectors and text recognition results; fusing lung image feature vectors and lung text feature vectors by using an attention mechanism through a lung fusion recognition model, and extracting image text fusion features for recognition to obtain a fusion recognition result; and obtaining a lung feature identification result through voting. The invention realizes accurate recognition of lung characteristics and improves the recognition accuracy and reliability. The method is suitable for the fields of intelligent medical treatment and the like, and can further promote the construction of intelligent cities.
Description
Technical Field
The invention relates to the field of artificial intelligence image classification, in particular to a lung feature identification method, a lung feature identification device, computer equipment and a storage medium.
Background
Under the current medical system, identification of lung characteristics mainly depends on manual judgment of lung image information by medical staff according to own experience, and because lung tissue movement is uneven and complex, time and energy of the medical staff are consumed in the judgment process, and the risk of judgment errors exists.
Disclosure of Invention
The invention provides a lung feature recognition method, a device, computer equipment and a storage medium, which realize the recognition by a lung feature recognition model comprising a lung image recognition model, a lung text recognition model and a lung fusion recognition model, and by applying an attention mechanism and combining a lung image to be recognized and lung text description to be recognized, realize the automatic, rapid and accurate recognition of lung features, improve the recognition accuracy and reliability and improve the recognition efficiency. The method is suitable for the fields of intelligent medical treatment and the like, and can further promote the construction of intelligent cities.
A method of lung feature identification, comprising:
acquiring data to be identified, wherein the data to be identified comprises a lung image to be identified and a lung text description to be identified;
inputting the data to be identified into a lung feature identification model, wherein the lung feature identification model comprises a lung image identification model, a lung text identification model and a lung fusion identification model;
performing lung image feature extraction on the lung image to be identified through the lung image identification model to generate a lung image feature vector and an image identification result, and performing lung text feature extraction on the lung text description to be identified through the lung text identification model to generate a lung text feature vector and a text identification result;
fusing the lung image feature vectors and the lung text feature vectors by using an attention mechanism through the lung fusion recognition model, and extracting and recognizing the fused features to obtain a fusion recognition result;
voting the image recognition result, the text recognition result and the fusion recognition result through the lung feature recognition model to obtain a lung feature recognition result corresponding to the data to be recognized; the lung feature recognition result indicates the lung feature category of the data to be recognized.
A lung feature identification device, comprising:
the system comprises a receiving module, a judging module and a judging module, wherein the receiving module is used for acquiring data to be identified, and the data to be identified comprises a lung image to be identified and a lung text description to be identified;
the input module is used for inputting the data to be identified into a lung feature identification model, and the lung feature identification model comprises a lung image identification model, a lung text identification model and a lung fusion identification model;
the first identification module is used for extracting lung image features of the lung image to be identified through the lung image identification model to generate a lung image feature vector and an image identification result, and extracting lung text features of the lung text description to be identified through the lung text identification model to generate a lung text feature vector and a text identification result;
the second identification module is used for fusing the lung image feature vector and the lung text feature vector by using an attention mechanism through the lung fusion identification model, and extracting and identifying the fused features to obtain a fusion identification result;
the voting module is used for voting the image recognition result, the text recognition result and the fusion recognition result through the lung feature recognition model to obtain a lung feature recognition result corresponding to the data to be recognized; the lung feature recognition result indicates the lung feature category of the data to be recognized.
A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, the processor implementing the steps of the lung feature identification method described above when executing the computer program.
A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the above-mentioned method of lung feature recognition.
According to the lung feature identification method, the lung feature identification device, the computer equipment and the storage medium, data to be identified are obtained; the data to be recognized comprises a lung image to be recognized and a lung text description to be recognized; inputting the data to be identified into a lung feature identification model comprising a lung image identification model, a lung text identification model and a lung fusion identification model; performing lung image feature extraction on the lung image to be identified through the lung image identification model to generate a lung image feature vector and an image identification result, and performing lung text feature extraction on the lung text description to be identified through the lung text identification model to generate a lung text feature vector and a text identification result; fusing the lung image feature vectors and the lung text feature vectors by using an attention mechanism through the lung fusion recognition model, and extracting and recognizing the fused features to obtain a fusion recognition result; voting is carried out on the image recognition result, the text recognition result and the fusion recognition result through the lung feature recognition model to obtain a lung feature recognition result corresponding to the data to be recognized, so that the lung image to be recognized is recognized through the lung image recognition model to obtain the image recognition result, the lung text description to be recognized is recognized through the lung text recognition model to obtain the text recognition result, then the image text fusion feature is extracted through the lung fusion recognition model to be recognized by combining the lung image to be recognized and the text description to be recognized by applying an attention mechanism to obtain the fusion recognition result, finally voting is carried out according to the image recognition result, the text recognition result and the fusion recognition result to obtain the lung feature recognition result, and the combination of the lung image to be recognized and the lung text description to be recognized is realized, the lung features are automatically, quickly and accurately identified through the lung feature identification model based on the multi-modal model, so that the identification accuracy and reliability are improved, and the identification efficiency is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments of the present invention will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to these drawings without inventive labor.
FIG. 1 is a schematic diagram of an application environment of a lung feature recognition method according to an embodiment of the present invention;
FIG. 2 is a flow chart of a method of lung feature identification in accordance with an embodiment of the present invention;
FIG. 3 is a flowchart of step S30 of a lung feature identification method according to an embodiment of the present invention;
FIG. 4 is a flowchart of step S30 of a lung feature identification method according to another embodiment of the present invention;
FIG. 5 is a flowchart illustrating the step S40 of the lung feature identification method according to an embodiment of the present invention;
FIG. 6 is a flowchart of step S50 of a lung feature identification method according to an embodiment of the present invention;
FIG. 7 is a functional block diagram of a lung feature recognition apparatus in accordance with an embodiment of the present invention;
FIG. 8 is a schematic diagram of a computer device in an embodiment of the invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The lung feature recognition method provided by the invention can be applied to the application environment shown in fig. 1, wherein a client (computer device) communicates with a server through a network. The client (computer device) includes, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, cameras, and portable wearable devices. The server may be implemented as a stand-alone server or as a server cluster consisting of a plurality of servers.
In an embodiment, as shown in fig. 2, a lung feature identification method is provided, which mainly includes the following steps S10-S50:
s10, acquiring data to be recognized, wherein the data to be recognized comprises a lung image to be recognized and a lung text description to be recognized.
Understandably, the lung image to be recognized is an image acquired by a lung shooting device, the lung shooting device may be selected according to requirements, for example, the lung shooting device is a CT device, an X-ray machine, or a three-dimensional projection device, the lung text description is a description of a lung feature in the lung image to be recognized, that is, the lung text description is complaint information for the lung image to be recognized, the lung feature is a feature embodied by lung tissue motion, for example, the lung feature includes a pleural depression feature, a tracheobronchial feature, a lung vacuole feature, a lung burr feature, a lung frosted glass-like feature, and the like, after the lung image to be recognized is acquired and the lung image to be recognized is input with the lung text description to be recognized, the lung image to be recognized and the lung text description to be recognized are determined as data to be recognized, triggering an identification request, wherein the identification request is a request for lung feature identification of the data to be identified, receiving the identification request, and acquiring the data to be identified in the identification request.
S20, inputting the data to be recognized into a lung feature recognition model, wherein the lung feature recognition model comprises a lung image recognition model, a lung text recognition model and a lung fusion recognition model.
Understandably, the lung feature recognition model is a multi-modal model after training, the lung feature recognition model can realize the recognition of the lung features of the data to be recognized, the lung feature recognition model comprises a lung image recognition model, a lung text recognition model and a lung fusion recognition model, the lung image identification model is characterized in that by extracting lung image features in the lung image to be identified, and the image recognition result is recognized by the image, and the lung image feature vector used for the lung fusion recognition model is generated, the lung image features are features of an image space embodied by lung tissue motion, the network structure of the lung image recognition model can be set according to the requirements of image recognition, for example, the network structure of the lung image recognition model is VGG16, VGG19, GoogleNet or ResNet, and the like, and preferably, the network structure of the lung image recognition model is the network structure of VGG 19; the lung text recognition model is used for recognizing a text recognition result by extracting lung text features in the lung text description to be recognized and performing text recognition, and generating lung text feature vectors for a lung fusion recognition model, wherein the lung text features are features of a text space embodied by lung tissue motion, a network structure of the lung text recognition model can be set according to requirements of language recognition, for example, the network structure of the lung text recognition model is TextCNN, LSTM or BERT, and the like, and preferably, the network structure of the lung text recognition model is selected from the network structure of TextCNN; the lung fusion recognition model is used for fusing the lung image feature vector and the lung text feature vector by applying an attention mechanism, extracting image text fusion features in the fused lung image feature vector and the lung text feature vector, and recognizing a fusion recognition result, wherein the image text fusion features are implicit features associated between the lung image feature vector and the lung text feature vector, namely global similarity features between the lung image feature vector and the lung text feature vector, a network structure of the lung fusion recognition model can be set according to requirements, for example, a network structure of the lung fusion recognition model is DenseNet, Deep LearningNet or LeNet, and the like, and preferably, the network structure of the lung fusion recognition model is a network structure of DenseNet.
In an embodiment, before the step S20, that is, before the step of inputting the data to be recognized into the lung feature recognition model, the method includes:
s201, a lung sample set is obtained, wherein the lung sample set comprises a plurality of lung samples, the lung samples comprise lung images and lung text descriptions associated with the lung images, and the lung samples are associated with a lung feature class label.
Understandably, the set of lung samples is a set of the lung samples, the lung samples are historically collected samples including lung images and lung text descriptions associated with the lung images, one of the lung samples is associated with a lung feature class label, the lung feature class label is a label labeled on the lung sample and related to a lung feature class, the lung images are historically collected images of lungs acquired by a lung photographing device, the lung text descriptions are descriptions of lung features in the lung images associated with the lung samples, and the lung feature class is a classification of the lung features, for example, the lung feature class includes a pleural depression feature class corresponding to a pleural depression feature, a tracheobronchial feature class corresponding to a tracheobronchial feature, and a lung vacuole feature class corresponding to a lung vacuole feature, Lung burr features corresponding to the lung burr features and lung frostlike features corresponding to the lung frostlike features.
S202, inputting the lung sample into a multi-modal model containing initial parameters; the multi-modal model comprises a lung sample image recognition model, a lung sample text recognition model and a lung sample fusion recognition model.
Understandably, the multi-modal model identifies the characteristics of implicit relationship between an image and a text by measuring the image text matching similarity (global similarity between the image and the text), determines the classification result of the fusion of the image and the text, and comprises the initial parameters, the initial parameters comprise the parameters of the lung sample image identification model, the lung sample text identification model and the lung sample fusion identification model, and the parameters in the multi-modal identification models in other fields can be directly migrated into the initial parameters in the multi-modal model in a migration learning manner, so that the training process is simplified, the training time is shortened, and the training efficiency is improved The lung image recognition model is a trained lung sample image recognition model, the lung text recognition model is a trained lung sample text recognition model, and the lung fusion recognition model is a trained lung sample fusion recognition model.
S203, performing the lung image feature extraction on the lung image through the lung sample image recognition model to generate a lung sample image feature vector and an image sample recognition result, and performing the lung text feature extraction on the lung text description through the lung sample text recognition model to generate a lung sample text feature vector and a text sample recognition result.
Understandably, the lung image feature is a feature of an image space embodied by lung tissue motion, the lung sample image feature vector is a vector matrix with the lung image feature, the image sample identification result is a result of the lung sample image identification model identifying the lung feature in the lung image according to the similarity of the extracted lung image feature in the image space, the lung sample text feature vector is a vector matrix with the lung image feature, and the text sample identification result is a result of the lung sample text identification model identifying the lung feature in the lung text description according to the similarity of the extracted lung text feature in the text space.
S204, fusing the lung sample image feature vector and the lung sample text feature vector by the lung sample fusion recognition model through an attention mechanism, and learning and extracting the image text fusion feature and recognition to obtain a fusion sample recognition result.
Understandably, the lung sample image feature vector and the lung sample text feature vector are fused through the attention mechanism, and the learning extracts the image text fusion feature by capturing extraction of implicit similar features between an image and a text and local similarity measurement and extraction.
S205, voting is carried out on the image sample identification result, the text sample identification result and the fusion sample identification result to obtain a sample identification result.
S206, determining a loss value according to the sample identification result and the lung feature class label.
Understandably, the sample recognition result and the lung feature class label are input into a loss function of the multi-modal model, and the loss value is calculated through the loss function.
And S207, when the loss value does not reach a preset convergence condition, iteratively updating initial parameters of the multi-modal model until the loss value reaches the preset convergence condition, and recording the multi-modal model after convergence as a lung feature recognition model.
Understandably, the convergence condition may be a condition that the loss value is very small and does not decrease again after 6000 times of calculation, that is, when the loss value is very small and does not decrease again after 6000 times of calculation, the training is stopped, and the multi-modal model after convergence is recorded as the lung feature recognition model; the convergence condition may be a condition that the loss value is smaller than a set threshold value, that is, when the loss value is smaller than the set threshold value, the multi-modal model after stopping training and converging is recorded as a lung feature recognition model, and thus, continuously adjusting initial parameters in the multi-modal model when the loss value does not reach a preset convergence condition, triggering the lung sample image recognition model to perform the lung image feature extraction on the lung image to generate a lung sample image feature vector and an image sample recognition result, and simultaneously, the lung text feature extraction is carried out on the lung text description through the lung sample text recognition model, and a lung sample text feature vector and a text sample recognition result are generated, so that the lung sample text feature vector and the text sample recognition result can be continuously drawn to an accurate result, and the recognition accuracy is higher and higher. Therefore, the lung feature recognition of the multi-modal model can be optimized, and the accuracy and reliability of the lung feature recognition are improved.
S30, performing lung image feature extraction on the lung image to be identified through the lung image identification model to generate a lung image feature vector and an image identification result, and performing lung text feature extraction on the lung text description to be identified through the lung text identification model to generate a lung text feature vector and a text identification result.
Understandably, the lung image recognition model performs channel splitting and convolution on the lung image to be recognized, so as to extract the lung image features, the lung image features are features of an image space reflected by lung tissue motion, the lung image recognition model comprises a plurality of convolution layers, the convolution layers of the lung image recognition model can be marked as image convolution layers, the lung image to be recognized is convolved through the image convolution layers in the lung image recognition model according to different convolution kernels, so as to generate the lung image feature vectors corresponding to the image convolution layers, the lung image feature vectors are vector matrixes with the lung image features, the dimensionality of each lung image feature vector is different according to the different convolution layers, and the lung image recognition result is that the lung image recognition model recognizes the lung image features according to the extracted similarity of the lung image features in the image space As a result of the characterization, the lung text recognition model performs word vector conversion on the lung text description to be recognized, and then performs convolution to extract the lung text features, the lung text features are features of a text space embodied by lung tissue motion, the lung text recognition model includes a plurality of convolution layers, the convolution layers of the lung text recognition model can be marked as text convolution layers, the lung text description to be recognized is convolved according to different convolution kernels through each text convolution layer in the lung image recognition model to generate the lung text feature vectors corresponding to each text convolution layer, the lung text feature vectors are vector matrixes with the lung text features, the dimensionality of each lung text feature vector is different according to the difference of each text convolution layer, and the text recognition result is that the lung text recognition model performs text space correlation according to the extracted lung text features The similarity identifies the outcome of the lung features.
In an embodiment, as shown in fig. 3, in step S30, the performing, by the lung image recognition model, lung image feature extraction on the lung image to be recognized to generate a lung image feature vector and an image recognition result includes:
s301, splitting the lung image to be identified into a red channel image, a green channel image and a blue channel image through the lung image identification model; the lung image identification model is a network model constructed based on VGG 19.
Understandably, the lung image to be identified is an image of three channels, namely a red channel, a green channel, a blue channel and the like, namely the lung image to be identified comprises the red channel image corresponding to a red channel, the green channel image corresponding to a green channel and the blue channel image corresponding to a blue channel, splitting the lung image to be identified into the red channel image, the green channel image and the blue channel image through channel splitting, the red channel image is an image which represents the red degree of each pixel point through pixel values ranging from 0 to 255, the green channel image is an image representing the green degree of each pixel point by pixel values ranging from 0 to 255, the blue channel image is an image which represents the blue degree of each pixel point through pixel values ranging from 0 to 255.
The lung image identification model is a network model constructed based on VGG19, and the convolution depth of the lung image can be set to be 19, namely the network model with 19 levels of convolution layers.
S302, performing convolution extraction on the red channel image, the green channel image and the blue channel image respectively through the lung image identification model to obtain a red feature vector corresponding to the red channel image, a green feature vector corresponding to the green channel image and a blue feature vector corresponding to the blue channel image.
Understandably, the lung image recognition model convolves the red channel image to obtain the red characteristic vector, the red characteristic vector is used for extracting a vector reflected by a red space in the lung image characteristic, the lung image recognition model convolves the green channel image to obtain the green characteristic vector, the green characteristic vector is used for extracting a vector reflected by a green space in the lung image characteristic, the lung image recognition model convolves the blue channel image to obtain the blue characteristic vector, the blue characteristic vector is used for extracting a vector reflected by a blue space in the lung image characteristic, and the red characteristic vector, the green characteristic vector and the blue characteristic vector are determined to be the lung image characteristic vector.
S303, carrying out image recognition on the lung image feature vector through the lung image recognition model to obtain the image recognition result.
Understandably, the lung image feature vectors are subjected to image recognition through the lung image recognition model, the image recognition is that full-connection classification is carried out according to the extracted lung image feature vectors to obtain probability distribution of each lung feature category, and therefore the recognized image recognition result is output.
According to the lung image recognition method, the lung image to be recognized is split into a red channel image, a green channel image and a blue channel image through the lung image recognition model; the lung image identification model is a network model constructed based on VGG 19; performing convolution extraction on the red channel image, the green channel image and the blue channel image respectively through the lung image identification model to obtain lung image feature vectors; the lung image feature vectors are subjected to image recognition through the lung image recognition model to obtain the image recognition result, so that the lung image to be recognized is split into a red channel image, a green channel image and a blue channel image, the network model constructed based on VGG19 is subjected to convolution on each channel image to extract lung image features to obtain the lung image feature vectors, the image recognition result is output according to the lung image feature vectors, the lung image features in the lung image to be recognized can be extracted, the lung feature categories are recognized through the extracted lung image features, a data basis is provided for subsequent recognition, and the recognition accuracy and reliability are improved.
In an embodiment, as shown in fig. 4, in step S30, the performing lung text feature extraction on the to-be-recognized lung text description through the lung text recognition model to generate a lung text feature vector and a text recognition result includes:
s304, segmenting the lung text description to be recognized through the lung text recognition model, and constructing a text word vector corresponding to the lung text description to be recognized, wherein the lung text recognition model is a network model constructed based on TextCNN.
Understandably, the word segmentation is to use a word dictionary to split the lung text description to be recognized into single words, the word dictionary contains word vectors corresponding to all medical terms and words related to the lung, then the split single words are converted into word vectors corresponding to the words, the word vectors can be converted in a word2vec or Glove conversion mode, and then the converted word vectors are spliced to be spliced into the text word vectors.
The lung text recognition model is a network model constructed based on TextCNN, namely the lung text recognition model has a network structure of TextCNN, the convolution depth of the lung text recognition model is set to be 19, namely the lung text recognition model has 19 levels of convolution layers, and the convolution depth of the lung text recognition model is the same as that of the lung image recognition model so as to facilitate the recognition of a subsequent lung fusion recognition model.
S305, performing channel expansion on the text word vector to generate a first text word vector, a second text word vector and a third text word vector.
Understandably, the channel expansion is a process of expanding the text word vector of a single channel to a vector matrix with a preset dimension and copying the vector matrix to a preset channel number, that is, expanding the text word vector to a vector matrix with the same dimension as the lung image feature vector, and the expansion mode can be set according to requirements, and the vector matrix is copied to a vector matrix with the same channel number as the lung image feature vector, so that the first text word vector, the second text word vector and the third text word vector which are the same as the vector matrix are obtained.
S306, performing convolution extraction on the first text word vector, the second text word vector and the third text word vector respectively through the lung text recognition model to obtain a first text feature vector corresponding to the first text word vector, a second text feature vector corresponding to the second text word vector and a third text feature vector corresponding to the third text word vector.
Understandably, convolving the first text word vector by the lung text recognition model to obtain the first text feature vector, convolving the second text word vector through the lung text recognition model to obtain the second text feature vector, convolving the third text word vector through the lung text recognition model to obtain the third text feature vector, wherein the convolution kernel to convolve the first text word vector, the convolution kernel to convolve the second text word vector, and the convolution kernel to convolve the third text word vector may be different, that is, lung text feature vectors are extracted from dimensions of different text spaces, and the first text word vector, the second text word vector and the third text word vector are determined as the lung text feature vectors.
S307, performing text recognition on the lung text feature vector through the lung text recognition model to obtain the text recognition result.
Understandably, the lung text feature vectors are subjected to text recognition through the lung text recognition model, the text recognition is that full-connection classification is carried out according to the extracted lung text feature vectors, the probability distribution of each lung feature category is obtained, and therefore the recognized text recognition result is output. The lung text recognition model is used for segmenting the lung text description to be recognized, and a text word vector corresponding to the lung text description to be recognized is constructed; the lung text recognition model is a network model constructed based on TextCNN; performing channel expansion on the text word vector to generate a first text word vector, a second text word vector and a third text word vector; performing convolution extraction on the first text word vector, the second text word vector and the third text word vector through the lung text recognition model to obtain lung text feature vectors, performing text recognition on the lung text feature vectors through the lung text recognition model to obtain text recognition results, so that the lung text feature vectors are obtained by performing word segmentation and text word vector construction on the lung text description to be recognized, performing channel expansion to generate the first text word vector, the second text word vector and the third text word vector, extracting lung text features through a network model constructed based on TextCNN, outputting the text recognition results according to the lung text feature vectors, extracting lung text features in the lung text description to be recognized, and recognizing lung feature categories through the extracted lung text features, and a data base is provided for subsequent identification, and the identification accuracy and reliability are improved.
S40, fusing the lung image feature vectors and the lung text feature vectors by the lung fusion recognition model through an attention mechanism, and extracting and recognizing the fused features to obtain a fusion recognition result.
Understandably, the attention mechanism is a mechanism learned by an additional feedforward neural network in neural network learning and identification through attention weight, by which an implicit relationship between the lung image feature vector and the lung text feature vector can be found, that is, the lung image feature vector and the lung text feature vector are weighted and fused according to the weight parameter corresponding to each convolution layer learned through the attention mechanism, so as to obtain the fused feature vector corresponding to each convolution layer, all the fused feature vectors are convolved, the image text fused feature is extracted, that is, the fused feature is extracted, the image text fused feature is an implicit feature associated between the lung image feature vector and the lung text feature vector, that is, a similarity feature between the lung image feature vector and the lung text feature vector is extracted, and identifying according to the extracted image text fusion characteristics, namely performing full-connection classification to obtain probability distribution of each lung characteristic category, thereby outputting the fusion identification result.
In an embodiment, as shown in fig. 5, in the step S40, that is, fusing the lung image feature vector and the lung text feature vector by the lung fusion recognition model using an attention mechanism, and extracting image text fusion features for recognition, obtaining a fusion recognition result, the method includes:
s401, weighting and fusing the lung image feature vectors and the lung text feature vectors by using an attention mechanism technology and through the weight parameters corresponding to the convolutional layers in the lung fusion recognition model to obtain fusion feature vectors corresponding to the convolutional layers.
Understandably, the attention mechanism technology is used for enhancing useful information in the feature vector, namely, performing weighted average on the weight parameters corresponding to each convolution layer according to the useful vector in the lung image feature vector and the lung text feature vector, and fusing to generate a fused feature vector corresponding to each convolution layer.
Wherein the convolution depth in the lung fusion recognition model is the same as the convolution depth of the lung image recognition model or the lung text recognition model, and the convolution depth in the lung fusion recognition model is preferably 19 levels.
In one embodiment, in step S401, the lung image feature vector includes a red feature vector, a green feature vector, and a blue feature vector; the lung text feature vector comprises a first text feature vector, a second text feature vector and a third text feature vector;
the lung image recognition model, the lung character recognition model and the lung fusion recognition model all have the same convolution levels, and convolution layers corresponding to the convolution levels are arranged in the three models;
the weighting and fusion of the lung image feature vector and the lung text feature vector through the weight parameters corresponding to the convolutional layers in the lung fusion recognition model to obtain fusion feature vectors corresponding to the convolutional layers comprises:
s4011, fusing the red feature vector and the first text feature vector corresponding to the same convolution level according to a first weight parameter corresponding to the convolution level to obtain a first fused feature vector.
Understandably, the red feature vector and the first text feature vector corresponding to the same convolution level are weighted according to a first weight parameter corresponding to the convolution level, that is, each component value in the red feature vector and the first text feature vector is weighted and averaged according to the first weight parameter, so as to obtain the first fused feature vector, and the dimensions of the red feature vector, the first text feature vector and the first fused feature vector are the same.
S4012, fusing the green feature vector and the second text feature vector corresponding to the same convolution level according to a second weight parameter corresponding to the convolution level to obtain a second fused feature vector.
Understandably, the green eigenvector and the second text eigenvector corresponding to the same convolution level are weighted according to a second weight parameter of the convolution level, that is, each component value in the green eigenvector and the second text eigenvector is weighted and averaged according to the second weight parameter, so as to obtain the second fused eigenvector, and the green eigenvector, the second text eigenvector and the second fused eigenvector have the same dimension.
And S4013, fusing the blue feature vector and the third text feature vector corresponding to the same convolution level according to a third weight parameter corresponding to the convolution level to obtain a third fused feature vector.
Understandably, the blue feature vector and the third text feature vector corresponding to the same convolution level are weighted according to a third weight parameter of the convolution level, that is, each component value in the blue feature vector and the third text feature vector is weighted and averaged according to the third weight parameter, so as to obtain the third fused feature vector, and the dimensions of the blue feature vector, the third text feature vector and the third fused feature vector are the same.
The execution sequence of steps S4011, S4012, and S4013 is not limited, and the steps may be executed serially or in parallel, and the first weight parameter, the second weight parameter, and the third weight parameter may be the same or different.
S4014, performing weighted average on the first fusion feature vector, the second fusion feature vector and the third fusion feature vector corresponding to the same convolution level to obtain the fusion feature vector.
Understandably, the weighted averaging is to average the first fused feature vector, the second fused feature vector and the third fused feature vector after weighting, and perform weighted averaging on the first fused feature vector, the second fused feature vector and the third fused feature vector corresponding to the same convolution level to obtain the fused feature vector corresponding to each convolution layer.
S402, extracting the image text fusion features of the fusion feature vector through the lung fusion recognition model.
Understandably, the extraction process of the image text fusion feature may be an extraction process of convolving the fusion feature vector of the first layer of the convolutional layer, then superimposing the fusion feature vector of the convolutional layer of the next layer of the convolutional layer to obtain a transfer feature vector, then convolving the transfer feature vector, continuously superimposing the transfer feature vector with the fusion feature vector of the convolutional layer of the next layer to obtain a transfer feature vector, and convolving the superimposed transfer feature vector until obtaining a one-dimensional feature vector.
And S403, recognizing according to the extracted image text fusion features through the lung fusion recognition model to obtain the fusion recognition result.
Understandably, the lung fusion recognition model carries out recognition according to the extracted image text fusion features, and the recognition is to obtain the probability distribution of each lung feature category according to the extracted image text fusion features, so as to output the recognized fusion recognition result.
The lung image feature vector and the lung text feature vector are subjected to weighted fusion by applying an attention mechanism technology and through the weight parameters corresponding to the convolutional layers in the lung fusion recognition model to obtain fusion feature vectors corresponding to the convolutional layers; extracting the image text fusion feature of the fusion feature vector through the lung fusion recognition model; the lung fusion recognition model is used for recognizing according to the extracted image text fusion features to obtain the fusion recognition result, so that useful information in images and texts can be enhanced by applying an attention mechanism, the global similarity between the images and the texts can be captured better, the lung image feature vectors and the lung text feature vectors are subjected to weighted fusion, the image text fusion features are extracted for recognition, and the accuracy and the reliability of lung feature recognition can be improved.
S50, voting the image recognition result, the text recognition result and the fusion recognition result through the lung feature recognition model to obtain a lung feature recognition result corresponding to the data to be recognized; the lung feature recognition result indicates the lung feature category of the data to be recognized.
Understandably, the voting is to perform weighted average on probability values corresponding to the same lung feature categories in the image recognition result, the text recognition result and the fusion recognition result, to finally determine a lung feature category with the highest probability value, and use the lung feature category with the highest probability value as the lung feature recognition result, where the lung feature recognition result includes the recognized lung feature category and the probability value corresponding to the category, the lung feature recognition result indicates the lung feature category of the data to be recognized, the lung features are features reflected by motion of lung tissues, for example, the lung features include pleural effusions, bronchotracheal features, lung vacuole features, lung burr features, lung glass-like features, and the like, and the lung feature category is a classification of the lung features, for example, the lung characteristic categories comprise pleural sunken characteristics corresponding to the pleural sunken characteristics, tracheobronchial characteristics corresponding to the tracheobronchial characteristics, lung vacuole characteristics corresponding to the lung vacuole characteristics, lung burr characteristics corresponding to the lung burr characteristics and lung frostlike characteristics corresponding to the lung frostlike characteristics.
The invention realizes the purpose of acquiring the data to be identified in the identification request; the data to be recognized comprises a lung image to be recognized and a lung text description to be recognized; inputting the data to be identified into a lung feature identification model comprising a lung image identification model, a lung text identification model and a lung fusion identification model; performing lung image feature extraction on the lung image to be identified through the lung image identification model to generate a lung image feature vector and an image identification result, and performing lung text feature extraction on the lung text description to be identified through the lung text identification model to generate a lung text feature vector and a text identification result; fusing the lung image feature vectors and the lung text feature vectors by using an attention mechanism through the lung fusion recognition model, and extracting image text fusion features for recognition to obtain a fusion recognition result; voting is carried out on the image recognition result, the text recognition result and the fusion recognition result through the lung feature recognition model to obtain a lung feature recognition result corresponding to the data to be recognized, so that the lung image to be recognized is recognized through the lung image recognition model to obtain the image recognition result, the lung text description to be recognized is recognized through the lung text recognition model to obtain the text recognition result, then the image text fusion feature is extracted through the lung fusion recognition model to be recognized by combining the lung image to be recognized and the text description to be recognized by applying an attention mechanism to obtain the fusion recognition result, finally voting is carried out according to the image recognition result, the text recognition result and the fusion recognition result to obtain the lung feature recognition result, and the combination of the lung image to be recognized and the lung text description to be recognized is realized, the lung features are automatically, quickly and accurately identified through the lung feature identification model based on the multi-modal model, so that the identification accuracy and reliability are improved, and the identification efficiency is improved.
In an embodiment, as shown in fig. 6, the voting, in the step S50, on the image recognition result, the text recognition result, and the fused recognition result by the lung feature recognition model, to obtain a lung feature recognition result corresponding to the data to be recognized includes:
s501, obtaining a weight parameter corresponding to the last convolution layer in the lung fusion recognition model.
Understandably, the weight parameters corresponding to the last layer of the convolutional layer in the lung fusion recognition model include image weights provided to the lung image feature vector corresponding to the last layer of convolutional layer and text weights of the lung text feature vector.
And S502, determining voting parameters according to the acquired weight parameters.
Understandably, the acquired image weight and the acquired text weight are kept unchanged, the image weight is used as a voting parameter of the image recognition result, the text weight is used as a voting parameter of the literary recognition result, and a numerical value one is used as a voting parameter of the fusion recognition result.
S503, voting is carried out on the image recognition result, the text recognition result and the fusion recognition result according to the voting parameters, and the lung feature recognition result is obtained.
Understandably, according to the voting parameters of the image recognition result, the voting parameters of the literary work recognition result, the text recognition result, the voting parameters of the fusion recognition result and the fusion recognition, the final probability distribution of each lung feature category is obtained through weighted average, and the lung feature category with the highest probability value is determined as the lung feature recognition result.
According to the invention, the weight parameters corresponding to the last convolution layer in the lung fusion recognition model are obtained; determining voting parameters according to the obtained weight parameters; and voting the image recognition result, the text recognition result and the fusion recognition result according to the voting parameters to obtain the lung feature recognition result, so that the lung feature category is finally recognized by objectively voting the image recognition result, the text recognition result and the fusion recognition result, and the accuracy and reliability of lung feature recognition are improved.
In one embodiment, a lung feature recognition device is provided, and the lung feature recognition device corresponds to the lung feature recognition method in the above embodiment one to one. As shown in fig. 7, the lung feature recognition apparatus includes a receiving module 11, an input module 12, a first recognition module 13, a second recognition module 14, and a voting module 15. The functional modules are explained in detail as follows:
the receiving module 11 is configured to receive an identification request and acquire data to be identified in the identification request; the data to be recognized comprises a lung image to be recognized and a lung text description to be recognized; the lung text description to be recognized is a description of lung features in the lung image to be recognized;
the input module 12 is used for inputting the data to be identified into a lung feature identification model; the lung feature recognition model comprises a lung image recognition model, a lung text recognition model and a lung fusion recognition model;
the first identification module 13 is configured to perform lung image feature extraction on the lung image to be identified through the lung image identification model to generate a lung image feature vector and an image identification result, and perform lung text feature extraction on the lung text description to be identified through the lung text identification model to generate a lung text feature vector and a text identification result;
the second identification module 14 is configured to fuse the lung image feature vectors and the lung text feature vectors by using an attention mechanism through the lung fusion identification model, and extract image text fusion features for identification to obtain a fusion identification result;
the voting module 15 is configured to vote the image recognition result, the text recognition result, and the fusion recognition result through the lung feature recognition model to obtain a lung feature recognition result corresponding to the data to be recognized; the lung feature recognition result indicates the lung feature category of the data to be recognized.
For the specific definition of the lung feature recognition device, reference may be made to the above definition of the lung feature recognition method, which is not described herein again. The modules in the lung feature recognition device can be wholly or partially implemented by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as shown in fig. 8. The computer device includes a processor, a memory, a network interface, and a database connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a method of lung feature recognition.
In one embodiment, a computer device is provided, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the computer program is executed by the processor, the lung feature recognition method in the above-mentioned embodiments is implemented.
In an embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which computer program, when being executed by a processor, is adapted to carry out the method of lung feature recognition according to an embodiment described above.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, databases, or other media used in embodiments provided herein may include non-volatile and/or volatile memory. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned function distribution may be performed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-mentioned functions.
The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present invention, and are intended to be included within the scope of the present invention.
Claims (10)
1. A method of lung feature identification, comprising:
acquiring data to be identified, wherein the data to be identified comprises a lung image to be identified and a lung text description to be identified;
inputting the data to be identified into a lung feature identification model, wherein the lung feature identification model comprises a lung image identification model, a lung text identification model and a lung fusion identification model;
performing lung image feature extraction on the lung image to be identified through the lung image identification model to generate a lung image feature vector and an image identification result, and performing lung text feature extraction on the lung text description to be identified through the lung text identification model to generate a lung text feature vector and a text identification result;
fusing the lung image feature vectors and the lung text feature vectors by using an attention mechanism through the lung fusion recognition model, and extracting and recognizing the fused features to obtain a fusion recognition result;
voting the image recognition result, the text recognition result and the fusion recognition result through the lung feature recognition model to obtain a lung feature recognition result corresponding to the data to be recognized; the lung feature recognition result indicates the lung feature category of the data to be recognized.
2. The lung feature recognition method of claim 1, wherein the performing lung image feature extraction on the lung image to be recognized through the lung image recognition model to generate a lung image feature vector and an image recognition result comprises:
splitting the lung image to be identified into a red channel image, a green channel image and a blue channel image through the lung image identification model, wherein the lung image identification model is a network model constructed based on VGG 19;
performing convolution extraction on the red channel image, the green channel image and the blue channel image through the lung image identification model to obtain a red feature vector corresponding to the red channel image, a green feature vector corresponding to the green channel image and a blue feature vector corresponding to the blue channel image;
and carrying out image recognition on the red feature vector, the green feature vector and the blue feature vector through the lung image recognition model to obtain the image recognition result.
3. The lung feature recognition method of claim 1, wherein the lung text feature extraction of the lung text description to be recognized through the lung text recognition model to generate a lung text feature vector and a text recognition result comprises:
segmenting the lung text description to be recognized through the lung text recognition model, and constructing a text word vector corresponding to the lung text description to be recognized, wherein the lung text recognition model is a network model constructed based on TextCNN;
performing channel expansion on the text word vector to generate a first text word vector, a second text word vector and a third text word vector;
performing convolution extraction on the first text word vector, the second text word vector and the third text word vector respectively through the lung text recognition model to obtain a first text feature vector corresponding to the first text word vector, a second text feature vector corresponding to the second text word vector and a third text feature vector corresponding to the third text word vector;
and performing text recognition on the first text feature vector, the second text feature vector and the third text feature vector through the lung text recognition model to obtain the text recognition result.
4. The lung feature recognition method of claim 1, wherein the fusing the lung image feature vector and the lung text feature vector by the lung fusion recognition model using an attention mechanism, and extracting and recognizing the fused features to obtain a fusion recognition result comprises:
weighting and fusing the lung image feature vectors and the lung text feature vectors by applying an attention mechanism technology and through the weight parameters corresponding to the convolutional layers in the lung fusion recognition model to obtain fusion feature vectors corresponding to the convolutional layers;
extracting the image text fusion feature of the fusion feature vector through the lung fusion recognition model;
and identifying according to the extracted image text fusion characteristics through the lung fusion identification model to obtain the fusion identification result.
5. The method of lung feature recognition of claim 4, wherein the lung image feature vectors include a red feature vector, a green feature vector, and a blue feature vector; the lung text feature vector comprises a first text feature vector, a second text feature vector and a third text feature vector;
the lung image recognition model, the lung character recognition model and the lung fusion recognition model all have the same convolution levels, and convolution layers corresponding to the convolution levels are arranged in the three models;
the weighting and fusion of the lung image feature vector and the lung text feature vector through the weight parameters corresponding to the convolutional layers in the lung fusion recognition model to obtain fusion feature vectors corresponding to the convolutional layers comprises:
fusing the red feature vector and the first text feature vector corresponding to the same convolution level according to a first weight parameter corresponding to the convolution level to obtain a first fused feature vector;
fusing the green eigenvectors and the second text eigenvectors corresponding to the same convolution level according to a second weight parameter corresponding to the convolution level to obtain a second fused eigenvector;
fusing the blue feature vector and the third text feature vector corresponding to the same convolution level according to a third weight parameter corresponding to the convolution level to obtain a third fused feature vector;
and carrying out weighted average on the first fusion characteristic vector, the second fusion characteristic vector and the third fusion characteristic vector corresponding to the same convolution level to obtain the fusion characteristic vector.
6. The lung feature recognition method of claim 4, wherein the voting of the image recognition result, the text recognition result and the fusion recognition result by the lung feature recognition model to obtain the lung feature recognition result corresponding to the data to be recognized comprises:
acquiring a weight parameter corresponding to the last convolution layer in the lung fusion recognition model;
determining voting parameters according to the obtained weight parameters;
and voting the image recognition result, the text recognition result and the fusion recognition result according to the voting parameters to obtain the lung feature recognition result.
7. The lung feature recognition method of claim 1, wherein the inputting the data to be recognized into the lung feature recognition model comprises:
acquiring a lung sample set, wherein the lung sample set comprises a plurality of lung samples, the lung samples comprise lung images and lung text descriptions associated with the lung images, and the lung samples are associated with a lung feature class label;
inputting the lung sample into a multimodal model containing initial parameters; the multi-modal model comprises a lung sample image recognition model, a lung sample text recognition model and a lung sample fusion recognition model;
performing lung image feature extraction on the lung image through the lung sample image recognition model to generate a lung sample image feature vector and an image sample recognition result, and performing lung text feature extraction on the lung text description through the lung sample text recognition model to generate a lung sample text feature vector and a text sample recognition result;
fusing the lung sample image feature vector and the lung sample text feature vector by using an attention mechanism through the lung sample fusion recognition model, and learning and extracting image text fusion features and recognition to obtain a fusion sample recognition result;
voting is carried out on the image sample identification result, the text sample identification result and the fusion sample identification result to obtain a sample identification result;
determining a loss value according to the sample identification result and the lung feature class label;
and when the loss value does not reach the preset convergence condition, iteratively updating the initial parameters of the multi-modal model until the loss value reaches the preset convergence condition, and recording the multi-modal model after convergence as a lung feature recognition model.
8. A lung feature identification device, comprising:
the system comprises a receiving module, a judging module and a judging module, wherein the receiving module is used for acquiring data to be identified, and the data to be identified comprises a lung image to be identified and a lung text description to be identified;
the input module is used for inputting the data to be identified into a lung feature identification model, and the lung feature identification model comprises a lung image identification model, a lung text identification model and a lung fusion identification model;
the first identification module is used for extracting lung image features of the lung image to be identified through the lung image identification model to generate a lung image feature vector and an image identification result, and extracting lung text features of the lung text description to be identified through the lung text identification model to generate a lung text feature vector and a text identification result;
the second identification module is used for fusing the lung image feature vector and the lung text feature vector by using an attention mechanism through the lung fusion identification model, and extracting and identifying the fused features to obtain a fusion identification result;
the voting module is used for voting the image recognition result, the text recognition result and the fusion recognition result through the lung feature recognition model to obtain a lung feature recognition result corresponding to the data to be recognized; the lung feature recognition result indicates the lung feature category of the data to be recognized.
9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor when executing the computer program implements a lung feature recognition method as claimed in any one of claims 1 to 7.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the lung feature identification method according to any one of claims 1 to 7.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010991495.4A CN111832581B (en) | 2020-09-21 | 2020-09-21 | Lung feature recognition method and device, computer equipment and storage medium |
| PCT/CN2021/096366 WO2022057309A1 (en) | 2020-09-21 | 2021-05-27 | Lung feature recognition method and apparatus, computer device, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010991495.4A CN111832581B (en) | 2020-09-21 | 2020-09-21 | Lung feature recognition method and device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111832581A true CN111832581A (en) | 2020-10-27 |
| CN111832581B CN111832581B (en) | 2021-01-29 |
Family
ID=72918738
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010991495.4A Active CN111832581B (en) | 2020-09-21 | 2020-09-21 | Lung feature recognition method and device, computer equipment and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN111832581B (en) |
| WO (1) | WO2022057309A1 (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112070069A (en) * | 2020-11-10 | 2020-12-11 | 支付宝(杭州)信息技术有限公司 | Method and device for identifying remote sensing image |
| CN113409306A (en) * | 2021-07-15 | 2021-09-17 | 推想医疗科技股份有限公司 | Detection device, training method, training device, equipment and medium |
| CN113469049A (en) * | 2021-06-30 | 2021-10-01 | 平安科技（深圳）有限公司 | Disease information identification method, system, device and storage medium |
| WO2022057309A1 (en) * | 2020-09-21 | 2022-03-24 | 平安科技（深圳）有限公司 | Lung feature recognition method and apparatus, computer device, and storage medium |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114863182A (en) * | 2022-05-23 | 2022-08-05 | 北京百度网讯科技有限公司 | Image classification method, and training method and device of image classification model |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107392224A (en) * | 2017-06-12 | 2017-11-24 | 天津科技大学 | A kind of crop disease recognizer based on triple channel convolutional neural networks |
| CN107665736A (en) * | 2017-09-30 | 2018-02-06 | 百度在线网络技术（北京）有限公司 | Method and apparatus for generating information |
| CN108334605A (en) * | 2018-02-01 | 2018-07-27 | 腾讯科技（深圳）有限公司 | File classification method, device, computer equipment and storage medium |
| CN108334843A (en) * | 2018-02-02 | 2018-07-27 | 成都国铁电气设备有限公司 | A kind of arcing recognition methods based on improvement AlexNet |
| CN108399409A (en) * | 2018-01-19 | 2018-08-14 | 北京达佳互联信息技术有限公司 | Image classification method, device and terminal |
| CN109543714A (en) * | 2018-10-16 | 2019-03-29 | 北京达佳互联信息技术有限公司 | Acquisition methods, device, electronic equipment and the storage medium of data characteristics |
| CN110188343A (en) * | 2019-04-22 | 2019-08-30 | 浙江工业大学 | Multi-modal emotion identification method based on fusion attention network |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111460889B (en) * | 2020-02-27 | 2023-10-31 | 平安科技（深圳）有限公司 | Abnormal behavior recognition method, device and equipment based on voice and image characteristics |
| CN111563551B (en) * | 2020-04-30 | 2022-08-30 | 支付宝(杭州)信息技术有限公司 | Multi-mode information fusion method and device and electronic equipment |
| CN111832581B (en) * | 2020-09-21 | 2021-01-29 | 平安科技（深圳）有限公司 | Lung feature recognition method and device, computer equipment and storage medium |
-
2020
- 2020-09-21 CN CN202010991495.4A patent/CN111832581B/en active Active
-
2021
- 2021-05-27 WO PCT/CN2021/096366 patent/WO2022057309A1/en active Application Filing
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107392224A (en) * | 2017-06-12 | 2017-11-24 | 天津科技大学 | A kind of crop disease recognizer based on triple channel convolutional neural networks |
| CN107665736A (en) * | 2017-09-30 | 2018-02-06 | 百度在线网络技术（北京）有限公司 | Method and apparatus for generating information |
| CN108399409A (en) * | 2018-01-19 | 2018-08-14 | 北京达佳互联信息技术有限公司 | Image classification method, device and terminal |
| CN108334605A (en) * | 2018-02-01 | 2018-07-27 | 腾讯科技（深圳）有限公司 | File classification method, device, computer equipment and storage medium |
| CN108334843A (en) * | 2018-02-02 | 2018-07-27 | 成都国铁电气设备有限公司 | A kind of arcing recognition methods based on improvement AlexNet |
| CN109543714A (en) * | 2018-10-16 | 2019-03-29 | 北京达佳互联信息技术有限公司 | Acquisition methods, device, electronic equipment and the storage medium of data characteristics |
| CN110188343A (en) * | 2019-04-22 | 2019-08-30 | 浙江工业大学 | Multi-modal emotion identification method based on fusion attention network |
Non-Patent Citations (1)
| Title |
|---|
| NICOLE_LIANG: "一、降维——机器学习笔记——降维（特征提取）", 《CSDN博客-HTTPS://BLOG.CSDN.NET/WEIXIN_39541558/ARTICLE/DETAILS/80053831》 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2022057309A1 (en) * | 2020-09-21 | 2022-03-24 | 平安科技（深圳）有限公司 | Lung feature recognition method and apparatus, computer device, and storage medium |
| CN112070069A (en) * | 2020-11-10 | 2020-12-11 | 支付宝(杭州)信息技术有限公司 | Method and device for identifying remote sensing image |
| CN113469049A (en) * | 2021-06-30 | 2021-10-01 | 平安科技（深圳）有限公司 | Disease information identification method, system, device and storage medium |
| CN113469049B (en) * | 2021-06-30 | 2024-05-10 | 平安科技（深圳）有限公司 | Disease information identification method, system, device and storage medium |
| CN113409306A (en) * | 2021-07-15 | 2021-09-17 | 推想医疗科技股份有限公司 | Detection device, training method, training device, equipment and medium |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2022057309A1 (en) | 2022-03-24 |
| CN111832581B (en) | 2021-01-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111832581B (en) | Lung feature recognition method and device, computer equipment and storage medium | |
| US11348249B2 (en) | Training method for image semantic segmentation model and server | |
| CN109241903B (en) | Sample data cleaning method, device, computer equipment and storage medium | |
| CN111310624B (en) | Occlusion recognition method, occlusion recognition device, computer equipment and storage medium | |
| WO2020228446A1 (en) | Model training method and apparatus, and terminal and storage medium | |
| CN112926654A (en) | Pre-labeling model training and certificate pre-labeling method, device, equipment and medium | |
| CN112330685A (en) | Image segmentation model training method, image segmentation device and electronic equipment | |
| CN112270686B (en) | Image segmentation model training method, image segmentation device and electronic equipment | |
| CN109285105A (en) | Method of detecting watermarks, device, computer equipment and storage medium | |
| CN111178162B (en) | Image recognition method, device, computer equipment and storage medium | |
| CN113192175A (en) | Model training method and device, computer equipment and readable storage medium | |
| CN113705685B (en) | Disease feature recognition model training, disease feature recognition method, device and equipment | |
| CN114330499A (en) | Method, device, equipment, storage medium and program product for training classification model | |
| CN114821736A (en) | Multi-modal face recognition method, device, equipment and medium based on contrast learning | |
| CN112818995A (en) | Image classification method and device, electronic equipment and storage medium | |
| CN114647713A (en) | Knowledge graph question-answering method, device and storage medium based on virtual confrontation | |
| CN111723870A (en) | Data set acquisition method, device, equipment and medium based on artificial intelligence | |
| CN109101984B (en) | Image identification method and device based on convolutional neural network | |
| CN112668462A (en) | Vehicle loss detection model training method, vehicle loss detection device, vehicle loss detection equipment and vehicle loss detection medium | |
| CN112241705A (en) | Target detection model training method and target detection method based on classification regression | |
| CN114764870A (en) | Object positioning model processing method, object positioning device and computer equipment | |
| CN112836682A (en) | Method and device for identifying object in video, computer equipment and storage medium | |
| CN113516182B (en) | Visual question-answering model training and visual question-answering method and device | |
| CN111582404A (en) | Content classification method and device and readable storage medium | |
| CN113705511A (en) | Gesture recognition method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |