CN111428072A - Ophthalmologic multimodal image retrieval method, apparatus, server and storage medium - Google Patents

Abstract

Embodiments of the present invention disclose an ophthalmology multimodal image retrieval method, device, server and storage medium. The method includes: acquiring a user's current eye image, where the eye image is an ophthalmology digital image, and the eye image is an ophthalmology digital image. The image contains multiple eye images of different modalities; the multiple different modal eye images are correspondingly input into multiple pre-trained deep learning models to fuse and train the multiple different modal eye images. Similarity and dissimilarity, and output multiple feature vectors, and generate a comparative analysis result according to the multiple feature vectors. By acquiring the user's unimodal eye image, the system performs recognition in the deep learning model and outputs the multimodal recognition result, which solves the problem that other potential eye diseases of the user cannot be found through the unimodal eye image in the prior art. The effect of retrieving other possible eye diseases through single-modal eye images is realized, which improves the user experience.

Description

Retrieval method, device, server and storage medium for ophthalmic multimodal images

technical field

Embodiments of the present invention relate to retrieval technologies, and in particular, to a retrieval method, device, server, and storage medium for ophthalmic multimodal images.

Background technique

With the development of imaging technology, ophthalmology digital images have become the main data of ophthalmology, and this trend drives the construction of ophthalmology image retrieval function to assist doctors in clinical decision-making. Traditionally, the ophthalmology image retrieval function adopts a text-based retrieval method. First, the image is described in text (to establish a correspondence between the text and the image), and a keyword query is entered during retrieval and the ranking result is returned. This method of "finding pictures by words" has semantic differences between the text description and the image content itself, which affects the retrieval effect. With the development of computer vision technology, Content-Based Image Retrieval (CBIR) method has been applied in ophthalmology. This method of "finding pictures by pictures" searches according to the color, shape, texture and other characteristics of the image itself, so as to avoid the semantic difference between text description and image content. Content-based Image Retrieval (CBIR) technology retrieves the most similar images based on the content of the images, and integrates information retrieval, computer vision and other technologies. In recent years, in the field of medical imaging, deep learning algorithms represented by deep convolutional networks (CNN) have achieved excellent performance in disease classification and lesion segmentation in ophthalmic images, and surpassed traditional classification in extracting features such as texture, color, and shape. It provides a technical basis for the construction of image retrieval function.

Diseases in various parts of the human body can be presented by lesions in the eye. Therefore, academia and industry are widely committed to automatically screening diseases by analyzing ophthalmic digital images through artificial intelligence algorithms, and relevant results have been published. For example, Baylor University in Texas has developed a free software called White Eye Detector that can identify eye cancer through photos, the University of Washington has developed a software BiliScreen that can diagnose liver cancer based on eye color, and China Zhizhen Health has launched intelligent fundus screening. camera. However, for automatic disease screening relying on ophthalmic medical imaging technology and image processing technology, its algorithms face challenges in interpretability and accuracy, and the samples used for training models also face problems such as difficulty in collection and subjective ambiguity in labeling. There is quite a long way to go. More importantly, although the computer screening results are only used as an auxiliary reference for doctors, the computer results will more or less affect the doctor's re-judgment, which may affect the final diagnosis.

SUMMARY OF THE INVENTION

The present invention provides an ophthalmology case retrieval method, device, server and storage medium, so as to achieve the effect of retrieving other potential eye problems that may exist through a single-modal eye image.

In a first aspect, an embodiment of the present invention provides a method for retrieving ophthalmic multimodal images, including: acquiring a current eye image of a user, the eye image is an ophthalmic digital image, and the eye image includes a variety of different Modal eye images;

Correspondingly input the eye images of the multiple different modalities into multiple pre-trained deep learning models to fuse and train the similarity and dissimilarity of the multiple different modal eye images and output multiple feature vectors , and generate a comparative analysis result according to the plurality of feature vectors.

Optionally, the ophthalmic digital image includes: fundus map, corneal nerve map and OCT map.

Optionally, the deep learning model is a multimodal convolutional neural network model.

Optionally, before inputting the single-modal eye image into the pre-trained deep learning model and outputting the multi-modal recognition result, the method further includes:

Label sample images with a variety of labels and build a database;

Building a multimodal convolutional neural network and using the sample images to train the multimodal convolutional neural network model to obtain a trained multimodal convolutional neural network model.

Optionally, marking the sample image with multiple labels and establishing a database includes:

If the sample image belongs to the same patient, it is marked as the first label;

If the sample image belongs to the same case, it is marked as the second label;

If the sample image is irrelevant, mark it as a third label;

A database is established of the sample images marked with the first label, the second label and the third label.

Optionally, the multimodal convolutional neural network model includes: a fundus map convolutional neural network model, a corneal neural map convolutional neural network model, and an OCT map convolutional neural network model.

Optionally, the comparative analysis results include: the most similar multiple sample images and their detailed case situations.

In a second aspect, an embodiment of the present invention further provides a retrieval device for ophthalmic multimodal images, the device comprising:

a data acquisition module, configured to acquire a current eye image of the user, the eye image is an ophthalmic digital image, and the eye image includes multiple eye images of different modalities;

A data recognition module for inputting the eye images of the various different modalities into a plurality of pre-trained deep learning models to fuse and train the similarity and dissimilarity of the eye images of the various different modalities And output multiple eigenvectors, and generate a comparative analysis result according to the multiple eigenvectors.

In a third aspect, an embodiment of the present invention further provides a server, where the server includes:

one or more processors;

storage means for storing one or more programs,

When the one or more programs are executed by the one or more processors, the one or more processors implement the method for retrieving ophthalmic multimodal images as described above.

In a fourth aspect, an embodiment of the present invention further provides a storage medium on which a computer program is stored, and when the program is executed by a processor, implements the retrieval method for an ophthalmic multimodal image as described above.

Embodiments of the present invention disclose an ophthalmology multimodal image retrieval method, device, server and storage medium, the method comprising:

Acquire the current eye image of the user, the eye image is an ophthalmological digital image, and the eye image includes multiple eye images of different modalities; the multiple different modal eye images are correspondingly input into multiple A pre-trained deep learning model fuses and trains the similarity and dissimilarity of the multiple different modal eye images, outputs multiple feature vectors, and generates a comparative analysis result according to the multiple feature vectors. The method for retrieving ophthalmic multimodal images provided by the embodiment of the present invention, by acquiring the user's single-modal eye image, the system performs recognition in the deep learning model, and outputs the multi-modal recognition result, which solves the problem that cannot be solved in the prior art. The problem of other potential eye diseases of the user is found through the single-modal eye image, and the effect of retrieving other possible eye diseases through the single-modal eye image is realized, which improves the user's experience.

Description of drawings

1 is a flowchart of a method for retrieving ophthalmic multimodal images according to Embodiment 1 of the present invention;

2 is a flowchart of a method for retrieving ophthalmic multimodal images according to Embodiment 2 of the present invention;

FIG. 3 is a schematic structural diagram of an ophthalmic multimodal image retrieval device according to Embodiment 3 of the present invention;

FIG. 4 is a schematic structural diagram of a server according to Embodiment 4 of the present invention.

Detailed ways

The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention. In addition, it should be noted that, for the convenience of description, the drawings only show some but not all structures related to the present invention.

Before discussing the exemplary embodiments in greater detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowchart depicts the steps as a sequential process, many of the steps may be performed in parallel, concurrently, or concurrently. Furthermore, the order of the steps can be rearranged. The process may be terminated when its operation is complete, but may also have additional steps not included in the figures. A process may correspond to a method, function, procedure, subroutine, subroutine, or the like.

Furthermore, the terms "first," "second," etc. may be used herein to describe various directions, acts, steps or elements, etc., but are not limited by these terms. These terms are only used to distinguish a first direction, act, step or element from another direction, act, step or element. For example, a first tag could be referred to as a second tag, and similarly, a second tag could be referred to as a first tag, without departing from the scope of this application. Both the first label and the second label are labels, but they are not the same label. The terms "first", "second" and the like should not be understood as indicating or implying relative importance or implying the number of technical features indicated. Thus, a feature defined as "first" or "second" may expressly or implicitly include one or more of that feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise expressly and specifically defined.

Example 1

FIG. 1 is a flowchart of a method for retrieving ophthalmic multimodal images provided by an embodiment of the present invention. This embodiment is applicable to a situation where a user searches for ophthalmic diseases online, and specifically includes the following steps:

Step 100: Acquire a current eye image of the user, where the eye image is an ophthalmological digital image, and the eye image includes multiple eye images in different modalities.

In this embodiment, the user's eye image uploaded or collected by the user is first obtained, and the eye image is an ophthalmological digital image. , OCT), Anterior Segment Optical Coherence Tomography (AS-OCT), Optical Coherence Tomography Angiography (OCTA), in vivo confocal microscopy (IVCM), fluorescein Fundus angiography (Fundus Fluorescein Angiography, FFA) and indocyanine green choroidal angiography (Indocyanine Green Angiography, ICGA) and so on. In this embodiment, the ophthalmic digital image includes: a fundus map, a corneal nerve map, and an OCT map. Imaging equipment outputs digital images by observing the morphology of blood vessels, nerves, cornea, crystals, lens, iris and other tissue structures in the human eye, including fundus mirror, slit lamp, and optical coherence tomography (OCT). The ophthalmic digital images generated by different imaging equipment are different, with different resolutions and different parts, and can be used for the diagnosis of different disease types. For example, optical coherence tomography of the posterior segment of the eye is of great value in the clinical examination and diagnosis of retinal diseases, macular diseases, optic nerve diseases, and glaucoma. In terms of data form, the digital image generated by each imaging method has one modality, and the multiple digital images generated by multiple imaging methods are multimodal.

Step 110, correspondingly input the eye images of the multiple different modalities into multiple pre-trained deep learning models to fuse and train the similarity and dissimilarity of the multiple different modal eye images, and output multiple images. feature vectors, and a comparative analysis result is generated according to the plurality of feature vectors.

In this embodiment, the deep learning model is a multi-modal convolutional neural network model, and by inputting eye images of multiple different modalities into multiple pre-trained deep learning models to fuse the training of the multiple different modalities The similarity and dissimilarity of eye images and output multiple feature vectors, and generate comparative analysis results according to the multiple feature vectors. In this embodiment, the comparative analysis results include: the most similar multiple sample images and their detailed In case of case, by returning the multimodal sample image and its case reasons, it can be provided to the user or doctor as a reference, which is convenient for judging whether the user may have other eye diseases that cannot be displayed by the single-modal eye image.

Exemplarily, some eye diseases sometimes do not appear alone, but occur simultaneously, and an elderly person often has several senile eye diseases. If you suffer from fundus lesions and cataracts at the same time, the fundus is generally examined by fundoscopy, and the cataract is examined by OCT, that is, different modalities of imaging methods are suitable for different diseases. When a patient presents with multiple eye lesions, multiple modalities of digital imaging may be required for diagnosis. It is of practical value to construct a similarity relationship between digital images of different modalities belonging to the same case. With case-level retrieval, the likelihood of multiple lesions across multiple modalities can be retrieved. In this case, multimodal retrieval is actually helpful for the discovery of multiple lesions. If it is known that a patient has disease A, in the multimodal retrieval function, the digital image retrieval of one modality of the patient is input, and digital images of other modalities are returned, and the returned digital images of other modalities are associated with B disease, the patient may have both diseases A and B at the same time. The value of retrieval lies in the fact that the requirements are partly clear and partly vague, and some vagueness will be continuously clarified through the retrieval results. For example, in the text search, if you want to find a book, you know the part of the title, but you don't know the full name. By searching for the part of the known book title, the search engine will return many similar results, and the similar results may exist in these similar results. That book, this time you know the full name of the book. Retrieval can help users constantly clarify their needs in the huge amount of information, and often there are unexpected results. Multimodal image retrieval may give you unexpected results in a variety of lesion scenarios, assisting doctors to further mine information to determine lesions.

Exemplarily, when a doctor makes a diagnosis, he extracts the patient's ophthalmic image for reading, and when it is difficult to make a decision on the image, he subconsciously searches for similar cases in the past for reference. At this time, if you only rely on the modality of the query image to retrieve, you may face insufficient cases of this modality (lack of difficult and miscellaneous diseases), or you may face similar cases retrieved and still cannot draw a diagnosis conclusion. At this time, you can further expand to other models. Retrieval of modalities, and use the results of retrievals from other modalities to assist in diagnosis. If a patient in a case has only one modal ophthalmic digital image, multimodal retrieval can be used to assist the doctor. However, if the patient has taken multiple modalities of ophthalmic digital images, although they can be retrieved one by one, the diagnosis process is time-consuming for doctors. When faced with intractable diseases, the patient only took one modal ophthalmic digital image, and the multimodal retrieval highlights its functional value. Even if the patient has taken a variety of ophthalmic digital images, the multi-modal retrieval can avoid multiple retrievals by the doctor in multiple modalities and improve the diagnosis efficiency.

This embodiment discloses a method for retrieving ophthalmic multi-modal images, the method includes: acquiring a current eye image of a user, the eye image is an ophthalmic digital image, and the eye image includes a variety of different modalities. Eye images; correspondingly input the eye images of the multiple different modalities into multiple pre-trained deep learning models to fuse and train the similarity and dissimilarity of the multiple different modal eye images and output A plurality of feature vectors, and a comparative analysis result is generated according to the plurality of feature vectors. The method for retrieving ophthalmic multimodal images provided by the embodiment of the present invention, by acquiring the user's single-modal eye image, the system performs recognition in the deep learning model, and outputs the multi-modal recognition result, which solves the problem that cannot be solved in the prior art. The problem of other potential eye diseases of the user is found through the single-modal eye image, and the effect of retrieving other possible eye diseases through the single-modal eye image is realized, which improves the user's experience.

Embodiment 2

FIG. 1 is a flowchart of a method for retrieving ophthalmic multimodal images provided by an embodiment of the present invention. This embodiment is applicable to a situation where a user searches for ophthalmic diseases online, and specifically includes the following steps:

Step 200: Acquire a current eye image of the user, the eye image is an ophthalmological digital image, and the eye image includes multiple eye images in different modalities.

Step 210 , mark the sample image with various labels and establish a database.

Specifically, step 210 includes: if the sample image belongs to the same patient, marking it as a first label;

If the sample image belongs to the same case, it is marked as the second label;

If the sample image is irrelevant, mark it as a third label;

A database is established of the sample images marked with the first label, the second label and the third label.

In this embodiment, the multimodal ophthalmic digital images are marked with three levels of labels, namely, the third label is irrelevant 0, the second label is case-related 1, and the first label is disease-related 2, and the correlation is getting higher and higher. Assuming that there are three modalities A, B, and C, namely, corneal nerve map, fundus map, and anterior segment OCT, the training samples are mainly labeled for these three modalities. If all three modalities belong to the same case (same patient), they are marked as 1, if they belong to the same disease, they are marked as 2, and if they are not related, they are marked as 0. If it is two modalities, label the digital images of both modalities. The generation of image-to-sample mainly considers the correlation of images between modalities. In theory, the higher the correlation, the more similar the learned hash codes.

Step 220 , establishing a multimodal convolutional neural network and using the sample image to train the multimodal convolutional neural network model to obtain a trained multimodal convolutional neural network model.

In this embodiment, the multimodal convolutional neural network model includes: a fundus map convolutional neural network model, a corneal neural map convolutional neural network model, and an OCT map convolutional neural network model. Loss function calculation is performed on the generated hash codes by training multiple neural network models.

Exemplarily, taking the two modalities of fundus map and OCT map as an example, the leftmost input image pairs of the two modalities (fundus map, OCT map), the middle network structure layer is the same, and the last layer is the Hash layer (generating K length of the hash code). During training, multi-modal loss training is performed according to the labels of image pairs between modalities. The loss function is designed in a discriminative manner, that is, the hash codes corresponding to images of different modalities belonging to the same label should have a smaller distance. A model trained based on a multimodal discriminative loss function, that is, a model fusion method. The network structure of different modalities is the same, but the weights are not shared (to avoid the influence of different modal features), but the final loss function integrates the correlation between the two, so that the images of different modalities have a certain similarity. After training, different modalities generate discrete hash representations for images according to their respective models, which is binarization.

During training, the output of the hash layer is between [-1, 1], and the tanh activation function is used; when the discrete hash code representation is generated after training, the hash layer is symbolized, that is, the sign symbol function is added to the tanh function. , so that K hash codes, each bit is either 1 or -1, and the distance can be calculated in Hamming space after binarization. In simple terms, when training, the K hash codes are values between [-1, 1] and are used for training the multimodal loss function. When generating hash codes for images based on the trained model, the K hash codes are the values of {-1,1}. The retrieval of lesion location is very important for case retrieval. Generally speaking, the discrimination of an image mainly depends on the identification and comparison of key areas (ie, lesion areas). Assuming that two images have a small proportion of the lesion area, if the entire image is compared, the feature representation of the lesion area will be ignored, which will lead to a similarity error. Simply put, in the whole image, the information of the lesion area is more weighted than other areas. In feature extraction, the model design should focus on the feature representation of the lesion area. Therefore, this scheme introduces a spatial attention mechanism into the model to focus on capturing the features of the lesion area. Based on the CNN model with spatial attention mechanism, high-dimensional images are mapped into low-dimensional hash codes. The model fusion method based on the multi-modal discriminative loss method builds the similarity of images between modalities, and the hash codes generated by the images with the same label should have a smaller distance.

The method based on multimodal model fusion generates hash codes for digital images, that is, supports Hamming distance calculation. The specific scenario is: input a new digital image, generate a series of K hash codes for the image through the same network, and then calculate the Hamming distance with the samples in the database that have generated hash codes. The smaller the distance, the greater the similarity. higher. Assuming that a model of OCT and fundus has been trained, and the samples in the database have been generated by the model hash code, now the doctor enters a fundus map to retrieve, this fundus map will first generate K through the model of fundus modalities A hash code, and the Hamming distance calculation is performed with the hash code of the sample in the database, and the most similar n results are returned. These n results may contain images of two modalities, because after model fusion, the digital images between the modalities establish a certain similarity, which is reflected in the K hash codes.

Step 230, correspondingly input the eye images of the various different modalities into a plurality of pre-trained deep learning models to fuse the similarity and dissimilarity of the eye images of the various different modalities, and output multiple data. feature vectors, and a comparative analysis result is generated according to the plurality of feature vectors.

This embodiment discloses a method for retrieving ophthalmic multi-modal images, the method includes: acquiring a current eye image of a user, the eye image is an ophthalmic digital image, and the eye image includes a variety of different modalities. Eye images; label sample images with various labels and establish a database; build a multimodal convolutional neural network and use the sample images to train the multimodal convolutional neural network model to obtain a trained The multi-modal convolutional neural network model correspondingly inputs the eye images of the various modalities into a plurality of pre-trained deep learning models to fuse and train the similarity and difference of the eye images of the various modalities. similarity and output multiple feature vectors, and generate a comparative analysis result according to the multiple feature vectors. The method for retrieving ophthalmic multimodal images provided by the embodiment of the present invention, by acquiring the user's single-modal eye image, the system performs recognition in the deep learning model, and outputs the multi-modal recognition result, which solves the problem that cannot be solved in the prior art. The problem of other potential eye diseases of the user is found through the single-modal eye image, and the effect of retrieving other possible eye diseases through the single-modal eye image is realized, which improves the user's experience.

Embodiment 3

The apparatus for retrieving ophthalmic multimodal images provided by the embodiments of the present invention can implement the retrieving methods for ophthalmic multimodal images provided in any embodiment of the present invention, and has functional modules and beneficial effects corresponding to the execution methods. FIG. 3 is a schematic structural diagram of an ophthalmic multimodal image retrieval apparatus 300 according to an embodiment of the present invention. Referring to FIG. 3 , the apparatus 300 for retrieving ophthalmic multimodal images provided by an embodiment of the present invention may specifically include:

The data acquisition module 310 is configured to acquire the current eye image of the user, the eye image is an ophthalmological digital image, and the eye image includes multiple eye images of different modalities;

The data identification module 320 is used for correspondingly inputting the eye images of the various modalities into a plurality of pre-trained deep learning models to fuse and train the similarity and dissimilarity of the eye images of the various modalities and output multiple eigenvectors, and generate a comparative analysis result according to the multiple eigenvectors.

Further, the ophthalmic digital image includes: fundus map, corneal nerve map and OCT map.

Further, the deep learning model is a multimodal convolutional neural network model.

Further, the acquiring the current eye image of the user, the eye image is an ophthalmological digital image, and the eye image includes multiple eye images of different modalities and further includes:

Label sample images with a variety of labels and build a database;

Building a multimodal convolutional neural network and using the sample images to train the multimodal convolutional neural network model to obtain a trained multimodal convolutional neural network model.

Further, the method of labeling the sample image with multiple labels and establishing a database includes:

If the sample image belongs to the same patient, it is marked as the first label;

If the sample image belongs to the same case, it is marked as the second label;

If the sample image is irrelevant, mark it as a third label;

A database is established of the sample images marked with the first label, the second label and the third label.

Further, the multimodal convolutional neural network model includes: a fundus map convolutional neural network model, a corneal neural map convolutional neural network model, and an OCT map convolutional neural network model.

Further, the comparative analysis results include: the most similar multiple sample images and their detailed case situations.

This embodiment discloses an ophthalmological multimodal image retrieval device, the device includes: a data acquisition module for acquiring a user's current eye image, the eye image is an ophthalmic digital image, and the eye image includes Eye images of a variety of different modalities; a data identification module for correspondingly inputting the eye images of the multiple different modalities into multiple pre-trained deep learning models to fuse and train the multiple different modalities of the eye images. The similarity and dissimilarity of the eye images and output a plurality of feature vectors, and a comparative analysis result is generated according to the plurality of feature vectors. The embodiment of the present invention provides an ophthalmological multimodal image retrieval device. By acquiring a user's single-modal eye image, the system performs recognition in a deep learning model, and outputs a multi-modal recognition result, which solves the problem that cannot be achieved in the prior art. The problem of other potential eye diseases of the user is found through the single-modal eye image, and the effect of retrieving other possible eye diseases through the single-modal eye image is realized, which improves the user's experience.

Embodiment 4

FIG. 4 is a schematic structural diagram of a computer server according to an embodiment of the present invention. As shown in FIG. 4 , the computer server includes a memory 410 and a processor 420. The number of processors 420 in the computer server may be one or more. In Fig. 4, a processor 420 is taken as an example; the memory 410 and the processor 420 in the device may be connected by a bus or in other ways, and the connection by a bus is taken as an example in Fig. 4 .

As a computer-readable storage medium, the memory 410 can be used to store software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to the method for retrieving ophthalmic multimodal images in the embodiments of the present invention (for example, an ophthalmic case The data acquisition module 310, the data identification module 320 in the retrieval device 300) the processor 420 executes various functional applications and data processing of the device/terminal/device by running the software programs, instructions and modules stored in the memory 410, That is, the retrieval method of the above-mentioned ophthalmic multimodal image is realized.

Wherein, the processor 420 is used for running the computer program stored in the memory 410, and realizes the following steps:

obtaining the current eye image of the user, the eye image is an ophthalmological digital image, and the eye image includes multiple eye images in different modalities;

Correspondingly input the eye images of the multiple different modalities into multiple pre-trained deep learning models to fuse and train the similarity and dissimilarity of the multiple different modal eye images and output multiple feature vectors , and generate a comparative analysis result according to the plurality of feature vectors.

In one of the embodiments, the computer program of the computer device provided by the embodiment of the present invention is not limited to the above method operation, and can also execute the retrieval method of the ophthalmic multimodal image provided by any embodiment of the present invention. related operations.

The memory 410 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal, and the like. Additionally, memory 410 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, memory 410 may further include memory located remotely from processor 420, and these remote memories may be connected to the device/terminal/device through a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

This embodiment discloses an ophthalmology multimodal image retrieval server, which is used to execute the following method. The method includes: acquiring a user's current eye image, where the eye image is an ophthalmology digital image, and the eye image includes Multiple different modal eye images; correspondingly input the multiple different modal eye images into multiple pre-trained deep learning models to fuse and train the similarity of the multiple different modal eye images sum dissimilarity and output multiple feature vectors, and generate a comparative analysis result according to the multiple feature vectors. The method for retrieving ophthalmic multimodal images provided by the embodiment of the present invention, by acquiring the user's single-modal eye image, the system performs recognition in the deep learning model, and outputs the multi-modal recognition result, which solves the problem that cannot be solved in the prior art. The problem of other potential eye diseases of the user is found through the single-modal eye image, and the effect of retrieving other possible eye diseases through the single-modal eye image is realized, which improves the user's experience.

Embodiment 5

Embodiment 5 of the present invention also provides a storage medium containing computer-executable instructions, where the computer-executable instructions are used to execute a method for retrieving ophthalmic multimodal images when executed by a computer processor, and the method includes:

obtaining the current eye image of the user, the eye image is an ophthalmological digital image, and the eye image includes multiple eye images in different modalities;

Correspondingly input the eye images of the multiple different modalities into multiple pre-trained deep learning models to fuse and train the similarity and dissimilarity of the multiple different modal eye images and output multiple feature vectors , and generate a comparative analysis result according to the plurality of feature vectors.

Of course, a storage medium containing computer-executable instructions provided by an embodiment of the present invention, the computer-executable instructions of which are not limited to the above-mentioned method operations, and can also execute an ophthalmic multi-modality method provided by any embodiment of the present invention. Related operations in the retrieval method of dynamic images.

The computer-readable storage medium of the embodiments of the present invention may adopt any combination of one or more computer-readable mediums. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples (a non-exhaustive list) of computer readable storage media include: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), Erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

A computer-readable signal medium may include a propagated data signal in baseband or as part of a carrier wave, with computer-readable program code embodied thereon. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device .

Program code embodied on a storage medium may be transmitted using any suitable medium, including - but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Computer program code for carrying out operations of the present invention may be written in one or more programming languages, including object-oriented programming languages—such as Java, Smalltalk, C++, but also conventional Procedural programming language - such as the "C" language or similar programming language. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or terminal. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (eg, using an Internet service provider through Internet connection).

This embodiment discloses a retrieval storage medium for ophthalmic multimodal images, which is used to execute the following method, the method includes: acquiring a current eye image of a user, the eye image is an ophthalmic digital image, and the eye image Including eye images of multiple different modalities; correspondingly inputting the multiple different modal eye images into multiple pre-trained deep learning models to fuse and train the similarity of the multiple different modal eye images and dissimilarity and output multiple feature vectors, and generate a comparative analysis result according to the multiple feature vectors. The method for retrieving ophthalmic multimodal images provided by the embodiment of the present invention, by acquiring the user's single-modal eye image, the system performs recognition in the deep learning model, and outputs the multi-modal recognition result, which solves the problem that cannot be solved in the prior art. The problem of other potential eye diseases of the user is found through the single-modal eye image, and the effect of retrieving other possible eye diseases through the single-modal eye image is realized, which improves the user's experience.

Note that the above are only preferred embodiments of the present invention and applied technical principles. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments, and can also include more other equivalent embodiments without departing from the concept of the present invention. The scope is determined by the scope of the appended claims.

Claims (10)

1. A method for searching multi-modal ophthalmology images is characterized by comprising the following steps:

acquiring a current eye image of a user, wherein the eye image is an ophthalmic digital image and comprises eye images in various different modes;

correspondingly inputting the eye images of the multiple different modes into a plurality of pre-trained deep learning models, fusing and training the similarity and the dissimilarity of the eye images of the multiple different modes, outputting a plurality of feature vectors, and generating a comparison analysis result according to the plurality of feature vectors.

2. The method for retrieving ophthalmic multimodal images as claimed in claim 1, wherein the ophthalmic digital image comprises: fundus, corneal nerve and OCT images.

3. The method as claimed in claim 1, wherein the deep learning model is a multi-modal convolutional neural network model.

4. The method for retrieving multi-modal ophthalmology video of claim 3, wherein the acquiring a current eye image of the user, the eye image being an ophthalmic digital video, the eye image including a plurality of eye videos of different modalities further comprises:

marking the sample image by using various labels and establishing a database;

and establishing a multi-modal convolutional neural network and training the multi-modal convolutional neural network model by using the sample image to obtain a trained multi-modal convolutional neural network model.

5. The method for retrieving multi-modal ophthalmology image of claim 4, wherein the labeling the sample image with a plurality of labels and establishing a database comprises:

if the sample images belong to the same patient, marking the sample images as a first label;

if the sample image belongs to the same case, marking the sample image as a second label;

if the sample image is not relevant, marking as a third label;

and establishing a database of sample images marked with the first label, the second label and the third label.

6. The method for retrieving multi-modal ophthalmology image of claim 4, wherein the multi-modal convolutional neural network model comprises: the eye fundus image convolution neural network model, the cornea neural image convolution neural network model and the OCT image convolution neural network model.

7. The method for retrieving multi-modal ophthalmic image of claim 1, wherein the comparative analysis result comprises: the most similar multiple sample images and their detailed case cases.

8. An ophthalmologic multi-modality image retrieval device, comprising:

the data acquisition module is used for acquiring a current eye image of a user, wherein the eye image is an ophthalmic digital image and comprises eye images in various different modes;

and the data identification module is used for correspondingly inputting the eye images in the plurality of different modes into a plurality of pre-trained deep learning models, fusing and training the similarity and the dissimilarity of the eye images in the plurality of different modes, outputting a plurality of feature vectors, and generating a comparison analysis result according to the plurality of feature vectors.

9. A server, characterized in that the server comprises:

one or more processors;

a storage device for storing one or more programs,

when executed by the one or more processors, cause the one or more processors to implement the method of retrieving ophthalmic multimodal imagery according to any one of claims 1 to 7.

10. A computer-readable storage medium on which a computer program is stored, the program, when executed by a processor, implementing the method for retrieving an ophthalmic multimodal image as claimed in any one of claims 1 to 7.

Images