KR20210154732A - Method for registrating fundus images to generate wide angle fundus image and device performing the same - Google Patents

Abstract

Embodiments relate to a method for registering a plurality of fundus to generate a wide-angle fundus image and to a device performing the same. The method includes the steps of: acquiring a plurality of fundus images obtained by photographing the eyeball of a subject, wherein at least one fundus image among the plurality of fundus images partially overlaps with at least one other fundus image; matching a first fundus image with a second fundus image of the plurality of fundus images; and generating a composite image in which at least a partial region of the first fundus image and at least a part of the second fundus image overlap, based on the re-registration result, to generate a wide-angle fundus image.

Description

A method for registering a plurality of fundus images to generate a wide-angle fundus image, and an apparatus for performing the same

The present invention relates to a technique for registering a fundus image, and more particularly, to a method of registering a plurality of fundus images to generate a wide-angle fundus image having a wider area than a photographing angle of a fundus camera, and an apparatus for performing the same .

The wide-angle fundus image is an image that integrates and visualizes diseases occurring in various areas of the eye at a glance, and is required for a comprehensive diagnosis of the subject's eye.

The wide-angle fundus image is acquired using a traditional montage technique, or is acquired using a wide-angle fundus imaging device (eg, a laser scan device manufactured by OPTOS).

The traditional montage technique creates a fundus image by registering a common area in a plurality of partial images of a retinal part in fragments. However, there is a problem in that it is affected by the existence of an obstacle other than the object to be photographed.

OPTOS's laser scanning technology, which extracts black and white data and visualizes it by overlaying an appropriate color image, can acquire a very wide area of 200 degrees up to 80% of the entire retina. However, errors such as motion artifacts may occur while the laser scans. In addition, there is a limitation in that the laser scanning device of OPTOS is required separately from the general fundus photographing device.

Korean Patent Registration No. 10-1761510 (2017.07.26.)

According to one aspect of the present invention, there may be provided an apparatus for matching a plurality of fundus images with a plurality of eye regions to generate a wide-angle fundus image having an area wider than a photographing angle of a fundus camera.

In addition, it is possible to provide a fundus image matching method for generating a wide-angle fundus image and a computer-readable recording medium recording instructions for performing the same.

A method of registering a plurality of fundus images to generate a wide-angle fundus image according to an aspect of the present application is performed by a computing device including a processor. The method includes: acquiring the plurality of fundus images obtained by photographing the eyeballs of the subject, wherein at least one fundus image of the plurality of fundus images partially overlaps with at least one other fundus image; matching a first fundus image of the plurality of fundus images with a second fundus image; and generating a synthesized image in which at least a partial region of the first fundus image and at least a part of the second fundus image overlap, based on the re-registration result, to generate the wide-angle fundus image.

In an embodiment, the step of matching the first fundus image and the second fundus image based on the feature points of the first and second fundus images includes at least through a feature descriptor for extracting features that are invariant to the size and rotation of the image. extracting one feature point from the first and second fundus images; sampling at least one of the feature points extracted from the first and second fundus images; and rigid body registration of the first and second fundus images based on the sampled feature points.

In an embodiment, the generating of the first blood vessel mask from the first fundus image comprises extracting the shape of blood vessels of at least a portion of the first fundus image using a segmentation model based on region characteristics in the first fundus image. step; and generating a blood vessel mask having a shape corresponding to the blood vessel region. The segmentation model is a model learned based on the characteristics of a blood vessel region, and is learned to output blood vessels in an input image as a vessel probability map.

In an embodiment, the generating of the first blood vessel mask from the first fundus image may include: generating a binarized blood vessel mask by binarizing an image including the extracted blood vessel shape as a blood vessel region; detecting a valley between a blood vessel and an external boundary by inverting the image including the binarized blood vessel mask; and filling the inside of the valley so that the inside of the valley represents the blood vessel.

In an embodiment, the re-registering of the first fundus image and the second fundus image based on the first and second blood vessel masks comprises deformable registration of the first blood vessel mask and the second blood vessel mask. ) to re-register the first fundus image and the second fundus image.

In an embodiment, the generating of the wide-angle fundus image includes: based on the re-registration image, pixels of the second fundus image corresponding to the pixels of the first fundus image have the same intensity as the pixels of the first fundus image ( intensity) or transforming it to have a color; and generating a wide-angle fundus image including a region made of the modified pixels.

In an embodiment, the generating of the first blood vessel mask representing at least a portion of blood vessels in the first region may include, when a plurality of first fundus images are acquired, the plurality of first fundus images based on feature points of each first fundus image. registering a first fundus image of extracting a blood vessel shape of at least a portion of a first region from the first fundus image using a segmentation model based on region characteristics within the fundus image; and generating a first blood vessel mask based on the extracted blood vessel shapes of the plurality of first fundus images.

In an embodiment, the generating of the first blood vessel mask based on the extracted blood vessel shapes of the plurality of first fundus images includes: non-rigid body registration of the extracted blood vessel shapes of the plurality of first fundus images; and generating the first blood vessel mask based on the matched plurality of blood vessel shapes.

In an embodiment, the non-rigid body matching may include matching in a direction in which pixel-wise similarity increases in the blood vessel probability maps of the first fundus images that are different from each other.

In one embodiment, the method includes: prior to registering the plurality of fundus images, detecting at least one of an optic disc and a fovea in the plurality of fundus images to thereby detect at least one of the optic disc and fovea. calculating one location; classifying the plurality of fundus images into a plurality of groups according to whether the optic nerve papilla and the fovea are detected; And with respect to at least one group, allocating a matching rank within the corresponding group to the fundus image for each group; may further include. The step of registering the first and second fundus images and generating a synthesized image is repeated until all fundus images included in each group are registered, and the first and second fundus images are registered and a synthesized image is generated. In the step, the first fundus image is an initial fundus image or an already generated synthetic image, and the second fundus image is a fundus image having a current matching order.

In an embodiment, the step of calculating the position by detecting at least one of the optic nerve head and fovea may include detecting at least one of the optic nerve head and fovea from the image frame of the fundus image input using a deep learning model. The deep learning model includes: a shared convolutional network including a plurality of convolutional layers and outputting a convolutional feature map; a regional designation network that generates an intermediate feature map by applying a sliding window method to the input convolutional feature map, and predicts a region of interest in which a candidate object is likely to be located based on the intermediate feature map; and an object detection network that includes a pooling layer and a fully connected layer and detects the optic disc or fovea in the region of interest based on a feature vector for the region of interest in the image frame initially input to the detection model.

In an embodiment, classifying the plurality of fundus images into a plurality of groups includes: classifying the fundus images in which both the optic nerve papilla and fovea are detected into a first group; classifying the fundus image in which only the optic nerve head is detected into a second group; classifying the fundus image in which only the fovea is detected into a third group; and classifying the fundus image in which neither the optic nerve head and the fovea are detected into a fourth group.

In an embodiment, the allocating the matching rank within the corresponding group to the fundus image for each corresponding group includes: For at least one of the first group to the third group, each reference fundus image for each group selecting; and allocating a matching rank within the corresponding group to the fundus image for each corresponding group using the reference fundus image. The first group matching order for the fundus images of the first group is assigned based on the distance between the optic nerve head of the first group reference fundus image and another fundus image in the first group, and the fundus image of the second group The matching ranking within the second group for the ? is assigned based on the distance between the optic nerve head of the reference fundus image of the second group and another fundus image in the second group, and within the third group for the fundus images of the third group The matching order may be assigned based on a distance between the optic disc of the reference fundus image of the third group and another fundus image in the third group.

In an embodiment, in the reference fundus image of the first group or the second group, the position of the optic nerve head detected in each fundus image may be the closest to the position of the center of the image frame.

In an embodiment, in the reference fundus image of the third group, the position of the detected fovea of each fundus image in the third group may be the closest to the position of the image center.

In an embodiment, the assigning of the intra-group matching rank to the fundus image for each corresponding group may include: allocating a fourth intra-group matching rank to the fourth group-specific fundus image. The matching rank within the fourth group is assigned based on the degree to which the feature points of each fundus image in the fourth group match the feature points of the first fundus image.

In an embodiment, the method may further include: setting a matching rank between groups. The matching order between the groups is set in the order of the first group, the second group, the third group, and the fourth group, or the order of the first group, the third group, the second group, and the fourth group.

A computer-readable recording medium readable by a computing device according to another aspect of the present application stores program instructions operable by the computing device. When the program instruction is executed by the processor of the computing device, the processor performs the fundus image matching method according to the above-described embodiments.

An apparatus according to another aspect of the present application includes: an image acquisition unit for acquiring a plurality of fundus images photographed by a fundus camera - At least one fundus image among the plurality of fundus images is partially with at least one other fundus image nested with ; The first fundus image and the second fundus image are matched based on the feature points of the first and second fundus images, a first blood vessel mask representing at least a part of blood vessels in the first region is generated, and a registration unit generating a second blood vessel mask representing at least a portion, and re-registering the first fundus image and the second fundus image based on the first and second blood vessel masks; and a synthesizer configured to generate a synthesized image in which at least a partial region of the first fundus image and at least a part of the second fundus image overlap, based on the re-registration result, to generate the wide-angle fundus image.

An apparatus according to another aspect of the present application includes: an image acquisition unit for acquiring a plurality of fundus images photographed by a fundus camera - At least one fundus image among the plurality of fundus images is partially with at least one other fundus image nested with ; Before registering the plurality of fundus images, at least one of an optic disc and a fovea is detected in the plurality of fundus images, and the plurality of fundus images are detected according to whether the optic nerve disc and fovea are detected. a classification unit for classifying into a plurality of groups, and assigning, to at least one group, a matching rank within the corresponding group to the fundus image for each corresponding group; For the fundus images included in each group, a matching unit that matches the fundus image having the current registration order with the synthesized image generated by matching at least one fundus image up to the previous registration order based on the registration order within the group ; And to generate a wide-angle fundus image, based on the registration result, the fundus image having the current registration order to at least one fundus image up to the previous registration order to register at least one fundus image to generate a new synthetic image from the synthesized unit comprises a You may.

The apparatus according to an aspect of the present invention may generate a wide-angle fundus image by registering a plurality of fundus images having different photographing ranges. As a result, the device can also support cross-sectional analysis.

In order to generate a visualization image such as a registered image for cross-sectional analysis, the device does not need to use a separate device such as a laser scanning device of OPTOS. In addition, the device may generate such a visualization image without being affected by the fundus imaging environment and/or imaging device.

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

In order to more clearly explain the technical solutions of the embodiments of the present invention or the prior art, drawings necessary for the description of the embodiments are briefly introduced below. It should be understood that the following drawings are for the purpose of explaining the embodiments of the present specification and not for the purpose of limitation. In addition, some elements to which various modifications such as exaggeration and omission have been applied may be shown in the drawings below for clarity of description.
1 is a conceptual diagram of an apparatus for performing a fundus image registration method for generating a wide-angle fundus image, according to an embodiment of the present invention.
2 shows a schematic network architecture of an optic disc/macular detection model, according to an embodiment of the present invention.
3 is a flowchart of a fundus image registration method for generating a wide-angle fundus image, according to an embodiment of the present invention.
FIG. 4 is a schematic diagram of a fundus input term matching method for generating the wide-angle fundus image of FIG. 3 .
5A to 5D are diagrams illustrating the results of post-processing the extracted vascular mask according to an embodiment of the present invention.
6A and 6B are diagrams illustrating a feature point-based matching result according to an embodiment of the present invention.
7A and 7B are diagrams illustrating a blood vessel mask-based matching result according to an embodiment of the present invention.
8A and 8B are diagrams illustrating a matched image from which color distortion is removed, according to an embodiment of the present invention.
9 and 10 show a result of generating a final wide-angle fundus image according to the method of FIG. 3 .

The terminology used herein is for the purpose of referring to specific embodiments only, and is not intended to limit the present invention. As used herein, the singular forms also include the plural forms unless the phrases clearly indicate the opposite. The meaning of "comprising," as used herein, specifies a particular characteristic, region, integer, step, operation, element and/or component, and includes the presence or absence of another characteristic, region, integer, step, operation, element and/or component. It does not exclude additions.

Although not defined otherwise, all terms including technical terms and scientific terms used herein have the same meaning as those commonly understood by those of ordinary skill in the art to which the present invention belongs. Commonly used terms defined in the dictionary are additionally interpreted as having a meaning consistent with the related technical literature and the presently disclosed content, and unless defined, are not interpreted in an ideal or very formal meaning.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

1 is a conceptual diagram of an apparatus for performing a fundus image registration method according to an embodiment of the present invention.

Referring to FIG. 1 , the device 10 is configured to acquire a fundus image from a fundus camera (not shown). The device 10 includes an image acquisition unit 100; feature point matching unit 300; and a blood vessel matching unit 500 .

The device 10 according to embodiments may be entirely hardware, entirely software, or may have aspects that are partly hardware and partly software. For example, the device may collectively refer to hardware equipped with data processing capability and operating software for driving the same. As used herein, terms such as “unit”, “module,” “device,” or “system” are intended to refer to a combination of hardware and software run by the hardware. For example, the hardware may be a data processing device including a central processing unit (CPU), a graphic processing unit (GPU), or another processor. In addition, software may refer to a running process, an object, an executable file, a thread of execution, a program, and the like.

The image acquisition unit 100 is configured to acquire a fundus image (or fundus photo) of the subject by photographing the subject's eyeball.

In certain embodiments, the fundus camera transmits photographing data of the subject's eye to the device 10 through wired/wireless electrical communication. The image acquisition unit 100 may be an interface for receiving photographing data from an external fundus camera in a wired/wireless manner. In some embodiments, the image acquisition unit 100 may be a wired interface port. In some other embodiments, the image acquisition unit 100 may be a communication module for wirelessly receiving data.

In other specific embodiments, a fundus camera may be integrated into the device 10 . In this case, the image acquisition unit 100 may be implemented as a fundus camera.

The fundus camera includes various fundus cameras capable of acquiring fundus images. For example, the fundus camera may include a mydriatic fundus camera, a non-mydriatic fundus camera, an OCT type fundus camera, and the like. The fundus image includes a retinal fundus image.

The image acquisition unit 100 may acquire a plurality of fundus images. The apparatus 10 generates a single composite image having a size wider than a single fundus image size by synthesizing the fundus images of at least some of the plurality of images acquired by the image acquisition unit 100 by partially matching each other.

The plurality of fundus images include a first group fundus image including both the optic nerve head and fovea in the image frame, a second group fundus image including only the optic nerve head within the image frame, a third group fundus image including only the fovea in the image frame, and A fourth group of fundus images that do not include both the optic nerve papilla and the fovea are included in the image frame.

Also, the image acquisition unit 10 may acquire data related to the fundus image. For example, the data related to the fundus image may include subject identification information (eg, name, identity information, identifier, etc.) capable of identifying the subject of the fundus image, a photographing time, and/or information related to a photographing device. . In addition, label data indicating whether the obtained fundus image is the fundus image of the left eye or the fundus image of the right eye may be further acquired. Then, the device 10 generates a wide-angle fundus image by matching fundus images for the same subject, or by matching fundus images of the eyes of the same side (eg, left eye fundus images or right eye fundus images) with each other It is also possible to generate a wide-angle fundus image. When using the pair of the first binocular fundus photo and the second binocular fundus photo taken at the second time point, the device 10 matches the first fundus photo of the left eye and the second fundus photo of the left eye, or the right eye Like matching the first fundus photo of , and the second fundus photo of the right eye, the apparatus 10 may perform a matching operation for each corresponding eye.

The image acquisition unit 100 supplies a plurality of fundus images to other components 200 , 400 , or 500 . Also, the image acquisition unit 100 may provide data related to the corresponding image while providing the fundus image.

The classifier 200 classifies the plurality of fundus images into first to fourth groups by detecting the optic nerve papilla or fovea in each image frame of the plurality of fundus images.

In an embodiment, the classifier 200 may detect the optic nerve head or fovea in the image frame using a pre-trained optic nerve head/macular detection model. Also, the classifier 200 may calculate the two-dimensional position of the detected object on the image frame. The classification unit 200 calculates the two-dimensional positions of the optic nerve papilla and fovea as pixel coordinates on the image frame.

2 shows a schematic network architecture of an optic disc/macular detection model, according to an embodiment of the present invention.

Referring to FIG. 2 , the optic nerve papilla/macular detection model (hereinafter, “detection model”) includes a shared convolutional network; regional network; It includes an object detection network.

The shared convolutional network includes a plurality of convolutional layers, the output of which is shared by a localized network and an object detection network.

The convolution feature map output by the last shared convolution layer of the shared convolutional network is input to the region-specific network.

The regional designation network generates an intermediate feature map by applying a sliding window method to the input convolutional feature map, and predicts a region of interest in which a candidate object is likely to be located based on the intermediate feature map.

The sliding method is a method of searching for a desired object for all ranges of input data while sliding a window on the input data. This window slides along the input convolutional feature map. The feature window at each sliding position is mapped to a lower-dimensional feature. Then, the region-specific network generates an intermediate feature map of a lower dimension than the input convolutional feature map. The regional designation network predicts a region of interest based on the intermediate feature map.

In an embodiment, the region-specific network may predict the ROI so that the candidate object (ie, the optic disc or fovea) is located in the center of the bounding box. Then, the position of the bounding box corresponds to the position of the center, that is, the position of the candidate object.

In some embodiments, the region-specific network may predict a region of interest using a bounding box having a fixed size in advance.

The size of the optic disc and fovea generally tends to be constant. The size of the bounding box may be preset to a value obtained based on this tendency.

As shown in FIG. 2 , the region-specific network predicts a region of interest that is likely to include the optic disc and fovea.

The object detection network includes a pooling layer and a fully connected layer.

The convolution feature map output by the last shared convolution layer of the shared convolutional network is also input to the object detection network. The input convolutional feature map is input to the pooling layer.

The pooling layer performs a pooling process on the input data based on the ROI prediction result by the regional designation network. The pooling layer is configured to output a feature vector when a feature map is input. Then, the feature vector for the region of interest in the image frame initially input to the detection model of FIG. 3 is input to the fully connected layer.

In some embodiments, when the size of the bounding box is fixed, the size of the intermediate feature map of the searched ROI is also fixed. Then, the object detection network acquires a feature vector of a fixed size and inputs it to the fully connected layer.

The fully connected layer is configured to predict the position of the optic nerve head and fovea by calculating the probability that the optic nerve head and fovea exist in the region of interest.

In an embodiment, the object detection network may output a result including the two-dimensional position and size of the bounding box on the input image frame, the class of each object (ie, the optic disc or fovea), and the confidence score for each class have. The confidence score may be a probability value or another conversion value based thereon.

The detection model detects the optic nerve head and fovea based on this result. The detection model detects the optic nerve head and fovea by selecting the region of interest having the highest value for the corresponding reliability for each class.

In one embodiment, the detection model selects the region of interest having the highest value for the corresponding reliability for each class, and when the reliability of the class of the selected region of interest is higher than a preset reliability threshold, the class of the selected region of interest It is also possible to detect an object of the optic nerve disc or fovea.

When the reliability is a probability value, the reliability threshold may be, for example, 0.9, but is not limited thereto.

The detection model is trained using a training data set including some or all of the first to fourth groups. The detection model is trained to have a parameter value for minimizing the difference between the actual value and the predicted value of the training fundus image.

The classification unit 200 automatically classifies the plurality of fundus images into first to fourth groups according to whether the optic nerve papilla or fovea is detected. The operation of the classification unit 200 will be described in more detail below with reference to FIG. 3 and the like.

The matching unit 400 may extract at least one feature point from the fundus image. The feature point is a point related to a geometric feature expressed in the fundus image. The geometrical features include edges, curves, and the like. The extracted key points may be used for a key point-based matching operation.

The matching unit 400 extracts a blood vessel region from each of a plurality of fundus images to match different fundus images. Also, the matching unit 400 may generate a blood vessel mask based on the extraction result of the blood vessel region.

The matching unit 400 may generate a blood vessel mask representing at least a portion of blood vessels in the fundus image.

The blood vessel mask has a blood vessel region in a mask shape. The blood vessel mask may include an outline of at least some blood vessels in the fundus image. In addition, the blood vessel mask may be configured to visually distinguish the outside of the blood vessel and the blood vessel region.

In an embodiment, the matching unit 400 may extract a blood vessel region from the fundus image using a pre-trained segmentation model. The segmentation model is configured to extract a blood vessel region based on region characteristics in the input fundus image. The segmentation model may calculate a vessel probability map including a probability value indicating whether a pixel constituting an image frame of the fundus image is a blood vessel, and determine a vessel region based on this.

The segmentation model is a machine learning model of a deep learning structure, comprising: a feature extraction layer; and a classification layer.

The feature extraction layer is configured to extract features used to classify a portion of an input image into a blood vessel class. When the segmentation model is a Convolutional Neural Network (CNN)-based model, the feature extraction layer may include a convolutional layer and/or a pooling layer.

The classification layer includes a classification layer configured to calculate a probability that a portion (eg, a pixel or a set of pixels) in an image is classified into a blood vessel class based on the extracted features. Then, for each pixel in the fundus image, a blood vessel probability map including the probability that the corresponding pixel is a blood vessel is calculated.

A blood vessel shape is determined based on the blood vessel probability map calculated by the segmentation model. In certain embodiments, a mask corresponding to a blood vessel shape based on the calculated blood vessel probability map may be generated as an eye blood vessel mask.

In some embodiments, the classification layer may be further configured to classify into a vascular class and a non-vascular class based on the calculated probability. In this case, the matching unit 400 may generate an image of a blood vessel region including pixels classified into blood vessel classes.

The segmentation model may have various structures based on a Convolutional Neural Network (CNN). The segmentation model may include a layer for extracting features from an input image; and a classification layer configured to calculate a probability that a portion (eg, a pixel) in an image is classified into a blood vessel class based on the extracted features. The segmentation model may include, for example, an SSANet, or a Rential SSANet-based structure, but is not limited thereto.

The segmentation model is trained to calculate a probability that a portion (eg, a pixel) in an image belongs to a blood vessel group by using a training sample including a plurality of training fundus images. The segmentation model may be a model learned by various learning processes having a class classification or calculation of a probability value for class classification of an input image part (eg, a pixel).

For example, parameters of a segmentation model having a fundus image as an input may be updated and learned in a direction in which a cost function of the corresponding model is minimized. Here, the cost function represents the difference between the result value calculated by the model and the actual result value. Learning these parameters is commonly referred to as optimization. The parameter optimization may include, for example, Adaptive Moment Estimation (ADAM), Momentum, Nesterov Accelerated Gradient (NAG), Adaptive Gradient (Adagrad), RMSProp, and various gradient descent schemes.

Through this learning process, the segmentation model is trained to enhance the ability to calculate higher probability values for actual blood vessels and lower probability values for non-vascular vessels in the fundus image. In addition, due to the enhancement of the probability value calculation capability, the blood vessel region identification capability may also be enhanced.

The matching unit 400 matches images (eg, a single image synthesized from the beginning to the previous registration order and a fundus image to be synthesized next) with each other based on the extracted features and/or blood vessels (ie, blood vessel mask). You may.

The synthesizing unit 500 may generate a single synthesized image from different matched fundus images.

The apparatus 10 may generate a wide-angle fundus image by the matching unit 400 and the synthesizing unit 500 . The wide-angle fundus image is implemented as a montage in which individual images are partially overlapped and synthesized. The creation of the montage starts from the central frame of the montage, and proceeds by synthesizing the fundus image into the previously created montage according to the registration order. Then, the size of the montage is gradually expanded to generate a final wide-angle fundus image having a comprehensive view of a plurality of fundus images.

The matching operation of the matching unit 400 and the wide-angle fundus image generation operation of the synthesizing unit 500 will be described in more detail below with reference to FIG. 3 and the like.

It will be apparent to one of ordinary skill in the art that the apparatus 10 may include other components not described herein. For example, the device 10 may include other hardware elements necessary for the operation described herein, including a network interface, an input device for data entry, and an output device for display, printing, or other data presentation. may be

The fundus image registration method for generating a wide-angle fundus image according to another aspect of the present invention may be performed by one or more processors like the above-described apparatus 10 . Hereinafter, for clarity of explanation, the steps of the fundus image registration method will be described in more detail based on embodiments performed by the apparatus 10 .

FIG. 3 is a flowchart of a fundus image registration method for generating a wide-angle fundus image according to an embodiment of the present invention, and FIG. 4 is a schematic diagram of a fundus input term matching method for generating the wide-angle fundus image of FIG. 3 . FIG. 4B is a schematic partial enlarged view of step S400 of FIG. 4A .

The fundus domain matching method (hereinafter, “matching method”) for generating the wide-angle fundus image includes: acquiring a plurality of fundus images (eg, by the image acquisition unit 100 ) ( S100 ). Each of the plurality of fundus images may include an optic nerve disc and/or a macular pericentre, or may not include all of them.

The registration method includes: (eg, by the classification unit 200 ), for each of a plurality of fundus images, detecting the optic nerve papilla and/or fovea and classifying them into a plurality of groups ( S200 ).

In an embodiment, the optic nerve head and/or fovea may be detected in the image frame of the fundus image using the pre-trained optic nerve head/macular detection model (S200). The detection model of step S200 and the detection operation of the optic nerve papilla and fovea using the same have been described above with reference to FIG. 2 , and detailed description thereof will be omitted.

In certain embodiments, the plurality of groups includes first through fourth groups. The step (S200) includes: classifying the fundus image in which both the optic nerve papilla and the fovea are detected into a first group; classifying the fundus image in which only the optic nerve head is detected into a second group; classifying the fundus image in which only the fovea is detected into a third group; and/or classifying a fundus image in which neither the optic nerve papilla nor the fovea is detected into a fourth group. For example, in step S200 , the plurality of images of step S100 may be classified into first to fourth groups, respectively.

In addition, the matching method includes: (eg, by the classification unit 200 ) allocating a matching rank among fundus images within a group for each group ( S300 ). The step (S300) may include: allocating a matching order to fundus images included in each of the partial or all groups with respect to some or all of the first to fourth groups; and/or allocating a matching order within the fourth group to the fundus images of the fourth group.

When a matching rank is assigned, the order of fundus images to be matched depends on the matching rank. For example, fundus images belonging to the first group are sequentially matched according to a matching order between fundus images in the first group. Alternatively, fundus images belonging to the fourth group may be sequentially registered according to the matching order among fundus images in the fourth group.

In one embodiment, the step (S300) may include: selecting, for at least one of the first group to the third group, each reference fundus image for each group; and allocating a matching rank within the corresponding group to the fundus image for each corresponding group using the reference fundus image. For example, a reference fundus image for each of the first to third groups is selected, and a matching rank within the group is assigned using the reference fundus image for each group ( S300 ).

The fovea and the optic disc have close anatomical characteristics. For this reason, since the fundus image in which the fovea is located in the central portion of the image frame generally also includes the optic disc, it can be treated as belonging to the first group.

In an embodiment, the reference fundus image of the first group may be selected as a fundus image having a minimum distance between the position of the optic nerve head detected for each fundus image and the central position of the image frame among fundus images belonging to the first group. There is (S300). When the first group of reference fundus images that may be used as the initial image frame of the wide-angle fundus image are selected in this way, as described above, it is advantageous to start with the center of the inside of the eyeball as a reference in generating the wide-angle fundus image as well as the advantage of visual This is because they can have all of the convenience of distinction.

In addition, the first group matching order for the fundus images of the first group may be assigned based on a distance between the optic nerve head of the reference fundus image of the first group and another fundus image in the first group. This is because, as described above, the optic nerve discs are more easily distinguished visually. As the distance between the optic nerve head of another fundus image and the optic nerve head of the reference fundus image is closer, a higher matching priority is assigned (S300).

The distance is, for example, ||p ^o _i -p ^o _ref || It may be calculated by Equation ( _2). Here, i is an index within the group, p ^o _i represents the pixel coordinates on the image frame of the optic nerve head of the i-th fundus image of the first group, and p ^o _ref is the pixel coordinate of the optic nerve head on the image frame of the reference fundus image. indicates

Since the fundus images of the first group and the second group include the optic nerve papilla, the reference fundus image of the first group may be used as the reference fundus image of the second group. That is, when the reference fundus image of the first group is selected, the reference fundus image of the second group is also automatically selected.

The matching order within the second group for the fundus images of the second group is assigned based on the distance between the optic nerve head of the reference fundus image of the second group and another fundus image in the second group (S300).

The reference fundus image of the third group is also similar to the process of selecting the reference fundus image of the first group described above, but is selected based on the position of the fovea instead of using the optic disc.

In an embodiment, the reference fundus image of the third group may be selected as a fundus image having a minimum distance between the position of the fovea detected for each fundus image and the central position of the image frame among fundus images belonging to the third group. There is (S300).

In addition, the intra-group matching order for the fundus images of the third group may be assigned based on a distance between the optic nerve head of the reference fundus image of the first group and another fundus image in the first group. This is because, as described above, the optic nerve discs are more easily distinguished visually. As the distance between the fovea of another fundus image and the fovea of the reference fundus image is closer, a higher matching priority is assigned (S300).

Meanwhile, the matching order among fundus images in the fourth group is assigned based on the matching degree of the feature points.

In one embodiment, the registration order among fundus images in the fourth group is already up to the initial fundus image (ie, the reference fundus image of the first group) or the previous matching rank for each feature point of the fundus images belonging to the fourth group. It may be assigned based on the degree of matching with the feature points of the synthesized image. A higher matching degree is assigned a higher matching rank.

In alternative embodiments, the matching method may further include: (eg, by the classification unit 200 ) setting a matching rank between groups (not shown).

When it is assumed that a wide-angle fundus image expressing the region inside the eyeball is generated, it is advantageous to match and synthesize the images based on the center of the eyeball. The fovea has anatomical characteristics close to the center of the retina. Therefore, the more the fovea is included, the higher the priority of registration should be. However, the fovea has properties that are obscure to distinguish visually, such as in the form of slightly darker dots within the retina. Therefore, it is not easy to use only the fovea as a reference in image alignment. On the other hand, the optic nerve papilla is relatively visually distinguishable compared to the fovea.

For this reason, as for the matching rank between the groups, the highest matching rank between the groups is assigned to the first group including both the fovea and the optic disc. The fourth group is assigned a matching rank among the lowest groups. A first group and a third group are assigned between them. For example, the order of the first group, the second group, the third group, and the fourth group or the order of the first group, the third group, the second group, and the fourth group may be set. Then, in order to generate a wide-angle fundus image, an initial fundus image is first selected from the first group, and the fundus images in the first group are sequentially matched to the initial fundus image according to the matching order of the fundus images in the first group. and create a composite image.

The step of setting the matching order among the groups may be performed before or after the step of acquiring a plurality of fundus images for generating a wide-angle fundus image ( S100 ). For example, the step of setting the matching order between groups may be set before the step (S500) of synthesizing the matched fundus image, which will be described below.

Then, the overall matching order for all the plurality of fundus images of the first to fourth groups may be allocated. For example, if a matching rank is assigned to fundus images in the first group, the first ?? The fundus image may be sorted as having the next matching rank of the last fundus image having the lowest matching rank in the first group.

The registration method includes: (eg, by the registration unit 400 ) matching the start fundus image with the fundus image of the next registration order based on the registration order (S400).

In certain embodiments, when overall matching ranks for all the plurality of fundus images of the first to fourth groups are allocated, the fundus image added to the already synthesized image may be a single fundus image.

Step S400 will be described in more detail with reference to the reference fundus image of the first group having the highest matching rank and the fundus image of the next matching rank in the first group, which can be used as an initial montage. The source image of the montage, that is, the reference fundus image of the first group is referred to below as a first fundus image, and the fundus image of the next matching order is referred to as a second fundus image.

Referring to FIGS. 3 and 4B , the step S400 includes: generating a blood vessel mask of the first fundus image and the second fundus image ( S410 ). The blood vessel mask is generated by extracting the blood vessel region from the fundus image using a pre-trained segmentation model. Then, a first blood vessel mask representing at least a portion of blood vessels in the first region and a second blood vessel mask representing at least a portion of blood vessels in the second area are generated ( S410 ).

A blood vessel shape is determined based on the blood vessel probability map extracted in step S410 . In certain embodiments, a mask corresponding to a blood vessel shape based on the extracted blood vessel probability map may be generated as an eye blood vessel mask.

The operation of generating the blood vessel mask by extracting the division model and the blood vessel region in step S410 has been described above with reference to the operation of the matching unit 400 , and thus detailed description thereof will be omitted.

In alternative embodiments, the step ( S410 ) may include post-processing the blood vessel shape based on the extracted vessel probability map. Then, it is possible to generate a more accurate and improved vascular mask than the vascular mask based only on the vascular probability map calculated by the vascular model. Post-processing includes binarization and/or redefining the boundaries of vessel shape. In certain embodiments, the post-processing may be performed after feature point-based matching.

5A to 5D are diagrams illustrating results of post-processing the extracted blood vessel mask according to an embodiment of the present invention.

In the exemplary embodiment, the post-processing step may include: generating a binarized fundus mask by applying a hysteresis thresholding to the input image and binarizing it.

As shown in FIG. 5A , an image of a blood vessel region may be generated based on the blood vessel probability map calculated in step S410 . The double image technique is applied to the image of FIG. 5A.

The double threshold technique is a technique for determining target data using an upper threshold and a lower threshold. In order to prevent an error in which filamentray vessels appear cut off in the image, the vascular region of the input image is binarized based on a preset first threshold value τl and a second threshold value τh. The second threshold value τh may be a higher threshold value than the first threshold value τl. _{Pixels above the second threshold value τ h} are used as seeds for regions in which pixels with higher probability than the lower threshold value τ1 grow. The first threshold value and the second threshold value may be empirical values, such as _{τ 1} =0.1, τ _{h = 0.75, and the like.}

By this double-threshold technique, the boundary of the vessel shape is grown as a region of the pixel having a higher probability than the first threshold value. A binarized vascular mask may be generated based on the growth results.

In addition, the post-processing step may include: fine-tuning the blood vessel mask to align the vessel boundary with respect to the image gradient of the fundus image.

To this end, the post-processing step may include: detecting a valley between a blood vessel and an external boundary in a binarized image; and filling the inside of the valley. If the binarized image is inverted, the region between the blood vessel and the outer boundary is more clearly distinguished. For example, as shown in FIG. 5C , a plurality of regions (eg, a region between a blood vessel and an external boundary indicating a blood vessel inner, outer boundary, and/or blood vessel thickness, etc.) that were identically expressed as a blood vessel region in FIG. 5B are more clearly distinguished. In the inverted binarized image, a valley region between the blood vessel and the outer boundary is detected, and image processing is performed to fill the inside of the valley region. Then, as shown in FIG. 5D , a mask obtained by improving the initial blood vessel mask based on the blood vessel probability map may be generated as the final blood vessel mask. The process of filling the valley inside in step S410 may be performed, for example, through image erosion processing.

When these post-processing procedures are applied to the blood vessel shape calculated by the segmentation model in the first fundus image and/or the second fundus image, the more precisely post-processed blood vessel mask of the first fundus image and/or the second fundus image is is generated (S410).

In addition, the step (S400) may include: matching the first fundus image and the second fundus image based on the feature points of the first fundus image and the second fundus image (S430); and matching the first fundus image and the second fundus image based on the generated blood vessel mask (S450). Since the matching operation in step S450 is performed after the feature point matching in step S430, it is treated as a re-matching operation.

Step (S430) is a step of extracting a feature point in the fundus image; and matching the first fundus image and the second fundus image based on the extracted feature points.

The feature point includes a point that is easily identified while being distinguished from the surrounding background in matching the image. As described above, geometric feature points may be used as feature points. In an embodiment, the feature point may include a feature point related to the geometry of the blood vessel. For example, a feature point included in the shape of a blood vessel or an edge may be included.

In step S430, feature points may be extracted from the input image (eg, the first fundus image) through various feature extraction algorithms based on a feature descriptor. The feature extraction algorithm may include, for example, a Scale Invariaant Feature Trnasform (SIFT) that extracts a point having a maximum cornerity on an image axis and/or a scale axis based on Difference of Gaussian (DoG), but is limited thereto. it doesn't happen A feature point set including a plurality of feature points may be extracted in step S430 .

In step S430, the keypoint-based matching may be performed in the pixel domain.

The first fundus image and the second fundus image by rigid registration of the first feature point set extracted from the first fundus image and the second feature point set extracted from the first fundus image in the step S430 , respectively can also be matched.

In an embodiment, the step S430 may include matching the first feature point set and the second feature point set using a perspective transformation matrix. In some embodiments, the step S300 may further use the RANSAC algorithm to match the first set of key points and the second set of key points. For example, in the process of matching the first feature point set and the second feature point set using the perspective transformation matrix, a value that maximizes the degree of matching between the feature points of each set is calculated through the RANSAC algorithm to perform feature point based matching. may be

6A and 6B are diagrams illustrating a feature point-based matching result according to an embodiment of the present invention.

As shown in FIG. 6A , the first fundus image and the second fundus image are matched based on the feature points of blood vessels ( S430 ).

Referring to FIG. 6B , by matching feature points of blood vessels, a registered image in which the first fundus image and the second fundus image are matched, and visualized with the registration result of FIG. 6A may be acquired.

After the matching in step S430, the matching in step S450 is additionally performed.

In an embodiment, the matching method may further include: verifying the validity of the keypoint-based matching result after the keypoint-based matching ( S440 ). Further matching in step S450 is performed when validity is verified in step S440.

In an embodiment, for validation, the fundus image (eg, the second fundus image) having the next matching rank may be subjected to homography transformation ( S440 ). When the second fundus image is subjected to homography conversion, the feature points of the second fundus image are projected on another fundus image (ie, the first fundus image). With respect to the second fundus image, if the change before and after homography transformation does not satisfy the validity condition, the second fundus image is not registered and is not used to generate a wide-angle fundus image.

In an embodiment, the validity condition may be set to be satisfied when a difference in area between image frames before and after homography transformation is less than a preset threshold. The threshold value is a pixel area difference of an image frame before/after homography conversion, and may be an area change amount itself or an area change rate. When the area change rate is used as a threshold, the threshold may be set to a value of approximately 15% to 5%. For example, the threshold may be 10%. Then, when the difference between before and after homography transformation of the second fundus image is greater than 10%, the second fundus image is excluded from the target of additional registration and synthesis.

As a result, inaccuracies of registration, including the inappropriateness of applying 2D homography to represent the view transformation of a 3D spherical object, and failure of matching a limited number of feature points due to small overlap or insufficient texture, etc. The problem is at least partially resolved.

Referring again to FIGS. 3 and 4 , the first fundus image and the second fundus image are matched based on the blood vessel mask (or the post-processed blood vessel mask) in step S450 .

In an embodiment, for re-registration after the feature point-based registration of step S430, the first and second fundus images are re-registered by deformable registration of the first and second blood vessel masks. It can also be matched (S450).

Deformable registration is a non-linear form of registration in which a registration object can be freely deformed. For example, a matching target of a straight line may be deformable in a shape such as a curve, a rectangle, or a circle.

The transformation matching algorithm used in step S450 may include, for example, a B-Spline algorithm. The B-Spline algorithm is a matching algorithm that iteratively optimizes in the direction of increasing pixel similarity between two images. An optimized matching result is calculated by determining the parameter that has the highest similarity.

As described above in step S410 , when the first blood vessel mask and the second blood vessel mask are generated by the segmentation model, the deformation registration is performed based on the more precisely extracted blood vessel region.

For example, if the B-Spline algorithm is applied using the first blood vessel mask as the source shape and the second blood vessel mask as the target shape, one or more control points of the first blood vessel mask are extracted and the first blood vessel mask The shape of may be modified to have an optimization curve that maximizes the pixel similarity with the second blood vessel mask. Then, the first blood vessel mask and the second blood vessel mask may be re-registered so that errors present in the feature point-based registration result are finely adjusted.

7A and 7B are diagrams illustrating a blood vessel mask-based matching result according to an embodiment of the present invention.

Referring to FIG. 7A , a more sophisticated matching result may be obtained by performing deformation matching of the blood vessel mask in step S450 to fine-tune errors existing in the feature point-based matching result.

When only rigid body matching is performed, a portion of blood vessels may not be matched accurately (S430). For example, as shown in FIG. 6A , when only feature point-based matching is performed, as the distance from the optic nerve head increases, the matching may not be relatively precise. However, as shown in FIG. 7A , when deformation registration is further applied, the first fundus image and the second fundus image are non-rigidly registered, so that a precise registration result may be obtained ( S450 ).

Then, as shown in FIG. 4 , a registered image in which portions of the first fundus image and the second fundus image are overlapped with each other may be obtained. Also, as shown in FIG. 7B , a registered image that visualizes the result of matching the first fundus image and the second fundus image to be fine-tuned may be acquired.

A more accurate registered image may be obtained through two-stage registration consisting of such feature point-based registration and deformation registration.

In addition, the registration method includes a step ( S500 ) of synthesizing registered fundus images to generate a wide-angle fundus image (eg, by the synthesizing unit 500 ). Based on the registration result of step S450, the first fundus image and the second fundus image used for registration in step S400 are synthesized into a partially overlapping single image (S500). The synthesized single image has an image frame that is larger than the image frame of the synthesized first fundus image or the second fundus image. Due to this, a composite image having a wider view than a photographing area of a single fundus image is generated (S500).

As shown in FIG. 4 , the synthesized image in step S500 includes an overlapping portion of the first fundus image and the second fundus image and other portions.

In an embodiment, the step of synthesizing the first fundus image and the second fundus image for generating the wide-angle fundus image includes: after B-spline optimization, calculating a displacement vector from the registration of the vascular mask ; and applying the calculated displacement vector to the first fundus image or the second fundus image. The photomontage is expanded due to the application of the displacement vector. When the source image of the synthesis (eg, the first fundus image) and the target image of the synthesis (eg, the second fundus image) are specified, the displacement vector includes the source shape (eg, the first blood vessel mask) and the target shape (eg, the second fundus image). 2 may correspond to a parameter that minimizes the sum of distances between points of the blood vessel mask).

The displacement vector may be calculated through a search on a distance transform (DT) of the edge shape of the target image with respect to the edge shape of the source image. For example, a parameter that minimizes the sum of distances may be calculated by calculating the sum of distances at each point while moving the target shape of the target image in pixel units on the distance map with respect to the edge shape of the source image.

The synthesized image generated in step S500 may be provided to the user.

In an embodiment, the step S500 may include: minimizing a difference in image specification between the first fundus image and the second fundus image.

The image specification includes intensity, color, and/or contrast of an appearance presented in an image frame. Each image frame has individual image specifications due to changes in imaging factors affecting the fundus image, including color intensity, scattering, and eyelid opening of external light. Therefore, the image specification between the image to be additionally synthesized for montage (ie, the second fundus image) and the existing image (ie, the first fundus image) may be different. When fundus images having different image specifications are matched, overlapping regions have different pixel values. To solve this, the difference in image specifications is minimized (S500).

In an embodiment, the step of minimizing the difference in image specification between the first fundus image and the second fundus image includes: to remove color distortion around the overlapped region by registration, the first fundus image The method may include transforming the pixels of the second fundus image corresponding to the pixels of to have the same intensity or color.

The corresponding pixel relationship between the respective images is determined based on the pixel coordinates of the corresponding pixel of the registered image. In step S500, the corresponding pixel is determined based on the pixel coordinates of the re-registered image.

Then, a composite image including an overlapping region from which color distortion is removed, which is made of transformed pixels, may be generated as a single image.

In an embodiment, a difference in image specification between the first fundus image and the second fundus image may be minimized by using a multi-resolution spline technique in the overlap region and boundary of the first fundus image and the second fundus image. .

The multi-resolution spline technique may minimize the difference in image intensity between layers of an image frame by applying weights around the center of the pair of the first fundus image and the second fundus image, and then applying a Gaussian filter and a Laplacian filter. Then, as shown in FIG. 4 , a wide-angle fundus image from which a difference in image intensity is removed is obtained.

8A and 8B are diagrams illustrating a matched image from which color distortion is removed, according to an embodiment of the present invention.

In the feature point-based registered image, the registration part is the feature point part. Where a feature point of a blood vessel is used, the mating portion includes a blood vessel portion. When the color or intensity of the first and second fundus images is different, as shown in FIG. 8A , a registered image in which pixels around the matching portion are corrected to a single color or intensity may be obtained.

In the blood vessel-based registration image, the registration part is the extracted blood vessel region. When the color or intensity of the first and second fundus images is different, as shown in FIG. 8B , a registered image in which pixels around the matching portion are corrected to a single color or intensity may be obtained.

As a result of removing the difference in image specifications, it is possible to provide a wide-angle fundus image in which the matching portion has unity as a single image (S500).

The steps (S400) and (S500) are repeatedly performed for the fundus image having the next matching rank and the already generated synthetic image. As long as there is no fundus image that has not passed the validation, steps S400 and S500 are repeatedly performed according to the matching sequence for all the plurality of fundus images obtained in step S100.

For example, when the initial fundus image is selected, the remaining fundus images are synthesized according to the registration order in the fourth group from the registration order in the first group, and a final wide-angle fundus image is generated.

In alternative embodiments, another fundus image to be added to the already synthesized wide-angle fundus image may be a sub-wide-angle fundus image synthesized by matching fundus images in other groups. For example, when a first sub wide-angle fundus image is generated by matching fundus images in a first group and a second sub wide-angle fundus image is generated by registering fundus images in a second group, the first sub wide-angle fundus image and A more expanded wide-angle fundus image may be generated by registering the second sub-wide-angle fundus image.

In this case, the registration and synthesis of the first sub-wide-angle fundus image and the second sub-wide-angle fundus image may be performed through the entire sub-wide-angle fundus image, or may be performed through one fundus image among each of the sub-wide-angle fundus images. For example, when registering and synthesizing the first sub-wide-angle fundus image and the second sub-wide-angle fundus image, the first ?? The fundus image and the last fundus image having the lowest matching rank in the first group may be used.

9 and 10 illustrate a result of generating a final wide-angle fundus image according to the method of FIG. 3 .

9 and 10 are wide-angle fundus images generated from different sets of fundus images. 9 and 10 , according to the method of FIG. 3 , a wide-angle fundus image accurately matched in the entire area is generated.

According to the embodiments described above The operation by the apparatus and the fundus image matching method may be at least partially implemented as a computer program and recorded in a computer-readable recording medium. For example, embodied with a program product consisting of a computer-readable medium containing program code, which may be executed by a processor for performing any or all steps, operations, or processes described.

The computer may be any device that may be incorporated into or may be a computing device such as a desktop computer, laptop computer, notebook, smart phone, or the like. A computer is a device having one or more alternative and special purpose processors, memory, storage, and networking components (either wireless or wired). The computer may run, for example, an operating system compatible with Microsoft's Windows, an operating system such as Apple OS X or iOS, a Linux distribution, or Google's Android OS.

The computer-readable recording medium includes all types of recording identification devices in which computer-readable data is stored. Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage identification device, and the like. In addition, the computer-readable recording medium may be distributed in a network-connected computer system, and the computer-readable code may be stored and executed in a distributed manner. In addition, functional programs, codes, and code segments for implementing the present embodiment may be easily understood by those skilled in the art to which the present embodiment belongs.

Although the present invention as described above has been described with reference to the embodiments shown in the drawings, it will be understood that these are merely exemplary and that various modifications and variations of the embodiments are possible therefrom by those of ordinary skill in the art. However, such modifications should be considered to be within the technical protection scope of the present invention. Accordingly, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

In the fundus image registration method according to an aspect of the present invention, a) a blood vessel region may be extracted using machine learning, which is one of the fourth industrial technologies.

In detecting the fundus change, a wide-angle fundus image can be easily obtained using a fundus camera through the above-described process without requiring a separate laser scanning device. The fundus camera does not require invasive treatment, does not use radiation, and has a short imaging time of less than a few minutes. Fundus examination using such an fundus camera is very inexpensive (currently, 7,990 won for insurance, 20,770 won for general). In addition, fundus cameras are not only distributed to almost all ophthalmology clinics and ophthalmology hospitals, but also have a number of examination centers, so accessibility and dissemination are excellent.

If a wide-angle fundus image can be generated using fundus photography having these advantages, it is possible to support reading of various eye diseases related to fundus changes, and it is expected to achieve a breakthrough.

Claims (20)

A method of registering a plurality of fundus images to generate a wide-angle fundus image, performed by a computing device including a processor, the method comprising:
acquiring the plurality of fundus images obtained by photographing the eyeballs of the subject, wherein at least one fundus image among the plurality of fundus images partially overlaps with at least one other fundus image;
matching a first fundus image of the plurality of fundus images with a second fundus image; and
and generating a composite image in which at least a partial region of the first fundus image and at least a part of the second fundus image overlap, based on a re-registration result, to generate a wide-angle fundus image.
According to claim 1, wherein the step of matching the first fundus image and the second fundus image based on the feature points of the first and second fundus images,
extracting at least one feature point from the first and second fundus images through a feature descriptor for extracting features that are invariant to the size and rotation of the image;
sampling at least one of the feature points extracted from the first and second fundus images; and
Method comprising the step of rigid body registration of the first and second fundus images based on the sampled feature points.
The method of claim 1, wherein the generating of the first blood vessel mask from the first fundus image comprises:
extracting a blood vessel shape of at least a portion of the first fundus image using a segmentation model based on region characteristics in the first fundus image; and
creating a vascular mask having a shape corresponding to the vascular region;
The split model is
As a model learned based on the characteristics of the blood vessel region, a method characterized in that it is learned to output the blood vessels in the input image as a vessel probability map.
The method of claim 3, wherein the generating of the first blood vessel mask from the first fundus image comprises:
generating a binarized blood vessel mask by binarizing an image including the extracted blood vessel shape as a blood vessel region;
detecting a valley between a blood vessel and an external boundary by inverting the image including the binarized blood vessel mask; and
and filling the interior of the valley such that the interior of the valley represents the blood vessel.
The method of claim 1, wherein re-registering the first fundus image and the second fundus image based on the first and second blood vessel masks comprises:
and re-registering the first fundus image and the second fundus image by deformable registration of the first blood vessel mask and the second blood vessel mask.
The method of claim 1, wherein the generating of the wide-angle fundus image comprises:
transforming a pixel of a second fundus image corresponding to a pixel of the first fundus image to have the same intensity or color as a pixel of the first fundus image based on the re-registration image; and
and generating a wide-angle fundus image including a region made of the deformed pixels.
The method of claim 1, wherein the generating of a first blood vessel mask representing at least a portion of a blood vessel in the first region comprises:
when a plurality of first fundus images are acquired, matching the plurality of first fundus images based on feature points of each first fundus image;
extracting a blood vessel shape of at least a portion of a first region from the first fundus image using a segmentation model based on region characteristics within the fundus image; and
and generating a first blood vessel mask based on the extracted blood vessel shapes of the plurality of first fundus images.
The method of claim 7, wherein the generating of the first blood vessel mask based on the extracted blood vessel shapes of the plurality of first fundus images comprises:
Non-rigid body registration of the extracted blood vessel shapes of the plurality of first fundus images; and
and generating the first vascular mask based on a plurality of matched vascular shapes.
The method of claim 8, wherein the non-rigid matching step comprises:
A method characterized in that matching is performed in a direction in which pixel-wise similarity increases in two different blood vessel probability maps of the first fundus image.
The method of claim 1 , wherein the method comprises:
calculating a position of at least one of the optic disc and fovea by detecting at least one of an optic disc and a fovea from the plurality of fundus images before matching the plurality of fundus images;
classifying the plurality of fundus images into a plurality of groups according to whether the optic nerve papilla and the fovea are detected; and
For at least one group, assigning a matching rank within the group to the fundus image for each group; further comprising,
The steps of registering the first and second fundus images and generating a composite image are repeated until all fundus images included in each group are registered,
In the step of registering the first and second fundus images and generating a synthesized image, the first fundus image is an initial fundus image or an already generated synthetic image, and the second fundus image is a fundus image having a current registration order. How to characterize.
The method of claim 10, wherein the step of calculating the position by detecting at least one of the optic nerve head and fovea comprises:
Detect at least one of the optic nerve papilla and fovea from the image frame of the fundus image input using a deep learning model,
The deep learning model is:
a shared convolutional network including a plurality of convolutional layers and outputting a convolutional feature map;
a regional designation network that generates an intermediate feature map by applying a sliding window method to the input convolutional feature map, and predicts a region of interest in which a candidate object is likely to be located based on the intermediate feature map; and
A method comprising a pooling layer and a fully connected layer and comprising an object detection network that detects the optic disc or fovea in the region of interest based on a feature vector for the region of interest in the image frame initially input to the detection model.
The method of claim 10, wherein classifying the plurality of fundus images into a plurality of groups comprises:
classifying the fundus images in which both the optic nerve papilla and the fovea are detected into a first group;
classifying the fundus image in which only the optic nerve head is detected into a second group;
classifying the fundus image in which only the fovea is detected into a third group; and
and at least one of classifying a fundus image in which both the optic nerve head and fovea are not detected into a fourth group.
13. The method of claim 12, wherein the allocating the matching rank within the group to the fundus image for each group comprises:
selecting each reference fundus image for each group with respect to at least one of the first to third groups; and
Allocating a matching rank within the group to the fundus image for each group by using the reference fundus image,
The first group matching order for the fundus images of the first group is assigned based on the distance between the optic nerve head of the first group reference fundus image and another fundus image in the first group,
The matching order within the second group for the fundus images of the second group is assigned based on the distance between the optic nerve head of the reference fundus image of the second group and another fundus image in the second group,
The third group matching order for the fundus images of the third group is assigned based on a distance between the optic nerve head of the third group reference fundus image and another fundus image in the third group.
14. The method of claim 13,
In the reference fundus image of the first group or the second group, the position of the optic nerve head detected in each fundus image is the closest to the position of the center of the image frame.
14. The method of claim 13,
In the reference fundus image of the third group, the position of the detected fovea of each fundus image in the third group is the closest to the position of the image center.
13. The method of claim 12,
Allocating the matching rank within the group to the fundus image for each group includes:
Allocating a matching rank within a fourth group to the fundus image for each fourth group,
The fourth group matching order is a method characterized in that the allocation based on the degree to which the feature points of each fundus image in the fourth group match the feature points of the first fundus image.
13. The method of claim 12,
Further comprising the step of setting a matching rank between groups,
The matching order between the groups is set in the order of the first group, the second group, the third group, and the fourth group, or the order of the first group, the third group, the second group, and the fourth group how to do it
18. A computer-readable medium readable by a computing device and storing program instructions operable by the computing device, wherein when the program instructions are executed by a processor of the computing device, the processor A computer-readable recording medium for performing a method of registering a plurality of fundus images to generate a wide-angle fundus image according to any one of claims.
an image acquisition unit for acquiring a plurality of fundus images taken by a fundus camera, wherein at least one fundus image among the plurality of fundus images partially overlaps with at least one other fundus image;
The first fundus image and the second fundus image are matched based on the feature points of the first and second fundus images, a first blood vessel mask representing at least a part of blood vessels in the first region is generated, and a registration unit generating a second blood vessel mask representing at least a portion, and re-registering the first fundus image and the second fundus image based on the first and second blood vessel masks; and
An apparatus comprising: a synthesizer configured to generate a composite image in which at least a partial region of a first fundus image and at least a part of the second fundus image overlap, based on a re-registration result, to generate a wide-angle fundus image.
an image acquisition unit for acquiring a plurality of fundus images taken by a fundus camera, wherein at least one fundus image among the plurality of fundus images partially overlaps with at least one other fundus image;
Before registering the plurality of fundus images, at least one of an optic disc and a fovea is detected in the plurality of fundus images, and the plurality of fundus images are detected according to whether the optic nerve disc and fovea are detected. a classification unit for classifying into a plurality of groups, and assigning, to at least one group, a matching rank within the corresponding group to the fundus image for each corresponding group;
For the fundus images included in each group, a matching unit that matches the fundus image having the current registration order with the synthesized image generated by matching at least one fundus image up to the previous registration order based on the registration order within the group ; and
In order to generate a wide-angle fundus image, based on the registration result, the fundus image having the current registration order is matched with at least one fundus image up to the previous registration order. Device characterized in that.

Images