JP7357955B2 - Image processing device and image processing method - Google Patents

Description

The present invention relates to an image processing device and an image processing method.

In recent years, development of medical devices using image processing has been actively conducted. Non-Patent Document 1 discloses a technique for extracting capillaries from an image using Rec-FCN (Recurrent Fully Convolutional Network) or the like.

Additionally, Patent Document 1 discloses a technique for improving the accuracy of classifying superficial blood vessels and body hair included in a three-dimensional image reconstructed from photoacoustic signals.

JP2019-025251A

Alvin I.Chen, Max L.Balter, Timothy J.Maguire and Martin L.Yarmush, Document title "Deep learning robotic guidance for autonomous vascular access", p.104-115, 2020

In addition to venules and arterioles, images of skin include body hair, dirt, and the like, and it is necessary to distinguish between venules and arterioles and body hair, dirt, and the like. However, even if the technique disclosed in the above-mentioned Non-Patent Document 1 is used, there is a possibility that venules and arterioles cannot be distinguished from body hair, dirt, and the like.

Furthermore, when the technique disclosed in Patent Document 1 is used, superficial blood vessels and body hair can be distinguished. However, the image processing device disclosed in Patent Document 1 needs to take a three-dimensional image, and there is a problem that a system including the image processing device becomes very expensive.

One aspect of the present invention aims to provide an image processing device and an image processing method that can extract microvessels (venules, arterioles) with high precision using a two-dimensional image.

In order to solve the above problems, an image processing device according to one aspect of the present invention includes a first region extraction section that extracts a body hair region from a two-dimensional image, and a first region extraction section that extracts a body hair region from the two-dimensional image. an image identification unit that identifies the body hair area extracted by the image identification unit; and a second area extraction unit that extracts a venule or arteriole area from the image in which the body hair area has been identified by the image identification unit. Be prepared.

Further, in order to solve the above problems, an image processing device according to one aspect of the present invention includes a first region extraction unit that extracts a body hair region from a two-dimensional image, and a first region extraction unit that extracts a body hair region from a two-dimensional image; a second region extracting section that extracts an artery region; and the body hair region extracted by the first region extracting section from an image of the venule or arteriole region extracted by the second region extracting section. and an image identification unit that identifies a region of the venule or the arteriole that is not.

Further, in order to solve the above problems, an image processing device according to one aspect of the present invention includes a snare portion tip extraction unit that extracts a snare portion tip from a two-dimensional image, and a snare portion tip extracting unit that extracts a snare portion tip from a two-dimensional image; a candidate region extracting section that extracts a candidate region of a vein or an arteriole; and a region of the venule or arteriole based on the snare tip and the candidate region extracted by the snare tip extracting section. and a microvessel determination unit that determines.

Further, in order to solve the above problems, an image processing method according to one aspect of the present invention extracts a body hair region from a two-dimensional image, and identifies the extracted body hair region from the two-dimensional image. , extracting a venule or arteriole region from the image in which the body hair region has been identified;

Furthermore, in order to solve the above problems, an image processing method according to one aspect of the present invention extracts a region of body hair from a two-dimensional image, and extracts a region of venules or arterioles from the two-dimensional image. , identifying a region of the venule or arteriole that is not the extracted body hair region from an image of the extracted region of the venule or arteriole;

Furthermore, in order to solve the above problems, an image processing method according to one aspect of the present invention extracts a snare tip from a two-dimensional image, and extracts a candidate region of a venule or arteriole from the two-dimensional image. is extracted, and the region of the venule or arteriole is determined based on the extracted tip of the snare portion and the candidate region.

According to one aspect of the present invention, it is possible to provide an image processing device and an image processing method that can extract microvessels (venules, arterioles) with high precision using a two-dimensional image.

1 is a diagram showing a configuration example of an image capturing device according to a first embodiment of the present invention. 1 is a diagram showing a configuration example of an image processing system according to a first embodiment of the present invention. 1 is a diagram showing a configuration example of an image processing apparatus according to a first embodiment of the present invention. FIG. 3 is a diagram illustrating an example of an image after being binarized by a preprocessing unit. FIG. 2 is a diagram for explaining an overview of U-Net. It is a learning curve diagram showing the correlation between the number of learning times and the Dice coefficient. FIG. 2 is a diagram for explaining an overview of CNN. FIG. 7 is a diagram showing an example of the correct answer rate for each object class (body hair, snare tip, microvessel). 3 is a flowchart for explaining the processing procedure of the image processing apparatus according to the first embodiment of the present invention. FIG. 3 is a diagram illustrating a configuration example of an image processing apparatus according to a second embodiment of the present invention. 7 is a flowchart for explaining a processing procedure of an image processing apparatus according to a second embodiment of the present invention. FIG. 7 is a diagram illustrating a configuration example of an image processing device according to a third embodiment of the present invention. FIG. 3 is a diagram illustrating a configuration example of a region extracting section. 12 is a flowchart for explaining a processing procedure of an image processing apparatus according to a third embodiment of the present invention. FIG. 3 is a diagram showing a region determined to be body hair by a region extraction unit. FIG. 7 is a diagram showing an image after microvessels have been identified by an image identifying unit.

(Embodiment 1)
<Background of the present invention>
The applicant has focused on the mechanism by which mosquito needles can suck blood from humans painlessly, and has been conducting research on a painless blood collection needle that imitates this mechanism. As a result of this research, we succeeded in creating the world's thinnest needle with an outer diameter of 0.09 mm and in blood collection using this needle. However, this needle cannot be used to collect blood from a blood vessel (with a diameter of about several mm), which is the conventional target for blood collection. This is because the length of the fine needle is limited to, for example, 2 mm in order to prevent breakage, and it cannot penetrate the outer skin of a large blood vessel.

For this reason, it is necessary to puncture and collect blood from small blood vessels (hereinafter referred to as microvessels), but these cannot be easily observed with the naked eye. Furthermore, due to diffused reflection on the skin surface, microvessels cannot be visually recognized using an ordinary microscope. Therefore, an image capturing device that visualizes microvessels is required. In this specification, microscopic blood vessels with a diameter of 0.1 mm to 0.2 mm called venules and arterioles that are distributed near the skin surface will be referred to as microvessels.

When observing the skin surface using a normal optical system, most of the irradiated light is diffusely reflected from the skin surface. As a result, it becomes difficult for the irradiated light to penetrate into the skin, and the visibility of microvessels decreases. Optical matching liquid is effective in solving this problem. The optical matching liquid is a liquid that has a refractive index close to that of the skin, and suppresses diffuse reflection of light by filling in the unevenness of the skin surface. The matching liquid developed by the applicant is a liquid (hereinafter referred to as hydrogel) that is a mixture of water, glycerin, hyaluronic acid, and chlorhexidine gluconate, and simultaneously disinfects the skin due to the effect of hexidine gluconate. It is possible to do so.

Furthermore, when a normal optical system is used, one of the factors that reduces the visibility of microvessels is that there is a small difference in white light absorption characteristics between microvessels and skin tissue. This is because white light includes light of various wavelengths, so the light absorption characteristics are also averaged. Therefore, irradiation with monochromatic light emphasizes microvessels and improves visibility.

<Example of configuration of image capturing device>
FIG. 1 is a diagram showing an example of the configuration of an image capturing device according to an embodiment of the present invention. As shown in FIG. 1, an image capturing device 2 for visualizing microvessels includes a camera 21, a green light source 22, and a slot 23.

The camera 21 is configured by a zoom microlens CMOS (Complementary Metal Oxide Semiconductor) camera or the like. Further, the green light source 22 is constituted by a green LED (Light Emitting Diode) or the like. The image capturing device 2 uses a green light source 22 with high visibility of microvessels, and suppresses diffuse reflection on the skin surface by applying a hydrogel 25 containing a disinfectant component to the skin surface. In this way, the image capturing device 2 makes it possible to capture highly accurate two-dimensional images of microvessels by preventing skin wrinkles from appearing in the image.

The movement of the blood sampling needle 24 is controlled by a control device (not shown) or the like so that microvessels are extracted by an image processing device, which will be described later, and blood is collected from the extracted microvessels through the slot 23.

<Arrangement of microvessels distributed near the skin surface (subepidermis)>
As shown in Figure 1, arteries distributed in the skin are called arterioles. From here, capillaries ascend in a loop shape, form a snare at the tip of the loop-shaped capillary, and transition into venules. Blood from the venules eventually flows into the cutaneous veins. In this specification, the tip portion of the snare shown in FIG. 1 (the uppermost portion: the portion closest to the skin surface) will be referred to as the snare tip.

<Example of configuration of image processing system 100>
FIG. 2 is a diagram illustrating a configuration example of an image processing system 100 according to an embodiment of the present invention. As shown in FIG. 2, the image capturing device 2 shown in FIG. 1 and the image processing device 1 according to the present embodiment are communicably connected via a communication network 3.

The image processing device 1 can acquire images including microvessels from the image capturing device 2 via the communication network 3. As an example of the communication network 3, a wireless LAN (Local Area Network), a wired LAN, a WAN (Wide Area Network), a public line network, a mobile data communication network, or a combination of these networks can be used.

<Configuration example of image processing device 1>
FIG. 3 is a diagram showing a configuration example of an image processing device 1 according to an embodiment of the present invention. The image processing device 1 includes an image acquisition section 11 , a preprocessing section 12 , a first region extraction section 13 , an image identification section 14 , and a second region extraction section 15 .

The image acquisition unit 11 acquires an image including microvessels from the image capturing device 2 via the communication network 3. The image acquisition unit 11 outputs the image acquired from the image capturing device 2 to the preprocessing unit 12.

<Operation of preprocessing section 12>
The preprocessing unit 12 adjusts the brightness and the like of the image through image processing, and then uses binarization processing to extract a region with brightness below a threshold value. The brightness and darkness of images in which microvessels are visualized by the image capturing device 2 are not constant within the image or from image to image. The preprocessing unit 12 performs normalization processing to equalize brightness values and adjust brightness and darkness in all images.

Specifically, since the image capturing device 2 visualizes microvessels using the green light source 22, the preprocessing unit 12 divides the color image into RGB and uses only Green (gray scale). . The preprocessing unit 12 determines an average brightness value from all images, and reduces unevenness in brightness of the entire image by making the brightness values determined for each small area in the image become the average value. . In this embodiment, the average value of brightness is assumed to be 128. Note that the brightness value is a value between 0 and 255.

With P(x0, y0) as the pixel of interest, the average value P(x, y) is determined within the peripheral small area P(125×125). Then, the preprocessing unit 12 calculates the luminance difference Pn(x,y) from the average value P(x,y) of the surrounding pixels using the following equations (Formula 1) and (Formula 2).

Pn (x, y) = P (x0, y0) - P (x, y) ... (Formula 1)
P(x0,y0)=Pn(x,y)×3+128...(Formula 2)
By adding 3 times the difference Pn(x,y) between the pixel of interest and the average value of the surrounding area to the specified average value of luminance 128 as in (Equation 2), the average value of the entire image becomes 128. Adjust the histogram so that

The preprocessing unit 12 performs binarization processing on the image after the normalization processing to extract a region having a brightness equal to or less than a threshold value. FIG. 4 is a diagram showing an example of an image after being binarized by the preprocessing unit 12. In FIG. 4, the black round part is the tip of the snare portion, which is the shape of the tip of the snare portion shown in FIG. 1 when viewed from above. Furthermore, the portions that overlap with the tips of the snare portions are microvessels. Note that in the center of FIG. 4, there is a region of body hair.

The first area extraction unit 13 is configured by, for example, U-Net, and is trained to extract areas of body hair. Further, the second area extraction unit 15 is configured by, for example, U-Net, and is trained to extract areas of microvessels. Note that machine learning is not limited to U-Net, and may be U-Net++, RNN, CNN, R-CNN, Fast R-CNN, Faster R-CNN, SegNet, etc.

<Learning method using U-Net>
FIG. 5 is a diagram for explaining the outline of U-Net. U-Net is classified as semantic segmentation in deep learning. Semantic segmentation refers to dividing an object into multiple regions pixel by pixel. U-Net is one of the FCNs (Fully Convolution Networks) without fully connected layers, and is a convolutional neural network with a U-shaped structure that performs high-speed and accurate image segmentation. Unlike a normal CNN, U-Net does not have a fully connected layer like FCN and SegNet, but is composed of convolutional layers. U-Net is a model composed of an encoder and a decoder.

As shown in FIG. 5, an encoder that performs downsampling is configured by four steps of 3×3 convolutional layers 131 to 134, and the number of convolutional channels is increased every time after maximum value pooling. Also, the feature map after maximum value pooling is combined with the feature map of the same spatial size of the Decoder.

In addition, a decoder that performs upsampling is configured by 4-step 3×3 convolutional layers 136 to 139, and the features from the encoder input through skip connections are combined to reduce the number of channels for each convolutional layer 136 to 139. Then, a probability map (segmentation map) of the same size as the input image is created.

When learning, first, a mask image to be used as U-Net teacher data is created from an image created by the preprocessing unit 12. The mask image is created by filling in areas (pixels) where microvessels exist from the image based on human judgment. The number of learning data is increased by subjecting the 26 mask images created by this method to rotation processing (up to 360° in 2° increments) and inversion processing (up, down, left and right). Through these processes, one original mask image can be increased to 360 images, so a dataset of 9360 images can be finally obtained. Note that a mask image for extracting body hair regions can also be created using a similar method. Further, the number of original mask images is just an example, and is not limited to this number.

To evaluate learning, the dataset is divided into 90% training data and 10% test data, and then learning is performed using U-Net. The Dice coefficient was used to confirm the detection accuracy of microvessel region extraction. The Dice coefficient evaluates the degree of similarity between the correct image and the guessed image of U-Net, and can be obtained by the following equation (Formula 3), where X is the correct region and Y is the guessed region.

FIG. 6 is a learning curve diagram showing the correlation between the number of learning times and the Dice coefficient. In FIG. 6, the horizontal axis represents the number of learning times, and the vertical axis represents the Dice coefficient. By learning while changing the combination of 90% training data and 10% test data, the learning curve diagram shown in FIG. 6 was obtained. As shown in FIG. 6, it can be seen that the Dice coefficient is almost saturated after 20 learnings. Based on this result, learning was performed 20 times for all datasets, and the first region extraction section 13 and the second region extraction section 15 were created. Note that the learning method and the number of times of learning are not limited to these.

<Learning method using CNN>
FIG. 7 is a diagram for explaining the outline of CNN. CNN is a representative method among deep learning, has many deep layers, and is a network that exhibits excellent performance particularly in the field of image recognition. The CNN is composed of an input layer 141, an output layer 147, convolution layers 142-144, a maximum pooling layer, and fully connected layers 145-146.

CNN automatically extracts necessary features from input data during learning. This function is realized by the convolutional layers 142 to 144 and the maximum pooling layer. The convolutional layers 142 to 144 are modeled after simple cells, and are capable of capturing features in response to a specific shape, and output a feature map by sliding a filter over the entire image. . The maximum pooling layer is used to reduce the dimensionality of the features extracted by the convolutional layer, reducing the number of parameters and the amount of calculation.

The pooling is maximum pooling with kernel size 2x2. In this embodiment, an object image cut out to 60x60 is input to CNN, and convolution with different filters and 2x2 pooling are repeated three times.

When learning, first, objects are manually cut out from the image created by the preprocessing unit 12 and classified into three types: body hair, microvessels, and snare tips, and are used as learning data. In this embodiment, body hair was classified into 154 hairs, snare tips were classified into 242 hairs, and microvessels were classified into 159 hairs. In this state, the total number of images is 555, which is a small amount of learning data, resulting in overlearning and low accuracy. Therefore, this learning data was rotated (in 30° increments up to 360°) to create a total of 6,660 images to increase the learning data.

To evaluate learning, the dataset was divided into 90% training data and 10% test data, and then CNN learning was performed 30 times. FIG. 8 is a diagram showing an example of the correct answer rate (precision rate, recall rate) for each object class (body hair, snare tip, microvessel).

In addition, in order to prevent body hair from being incorrectly determined as microvessels, as described later, even if an object is determined to be a microvessel, an object that does not include the snare tip is determined to be body hair. You can do it like this.

<Operations of the first region extracting section 13 and the second region extracting section 15>
When the second area extraction unit 15 extracts microvessels from the preprocessed image, body hair may be extracted as microvessels due to incorrect determination. Therefore, the first region extracting section 13 extracts body hair, and the second region extracting section 15 extracts microvessels from the image in which the body hair extracted by the first region extracting section 13 is identified. .

The first area extraction unit 13 extracts a body hair area from the image that has been preprocessed by the preprocessing unit 12 . The first region extraction unit 13 is trained to extract the body hair region using U-Net, CNN, etc., as described above.

The image identification section 14 identifies microvessel regions that are not body hair regions extracted by the first region extraction section 13 from the image preprocessed by the preprocessing section 12 . For example, the image identification section 14 may remove the image of the body hair region extracted by the first region extraction section 13 from the image preprocessed by the preprocessing section 12. Further, the image identification unit 14 may change the image so that the image of the body hair area extracted by the first area extraction unit 13 can be determined from the image preprocessed by the preprocessing unit 12. good. For example, the image identification unit 14 changes the color or gradation of each area in the image preprocessed by the preprocessing unit 12 to determine the area of microvessels and the area other than microvessels (body hair). (including areas) may be made distinguishable.

The second area extraction unit 15 extracts a capillary area from the image after the body hair area has been identified by the image identification unit 14. The second region extraction unit 15 is trained to extract microvessel regions using U-Net, CNN, etc., as described above.

<Processing procedure of image processing device 1>
FIG. 9 is a flowchart for explaining the processing procedure of the image processing apparatus 1 according to the first embodiment of the present invention. First, the image acquisition unit 11 acquires a two-dimensional image photographed by the image photographing device 2 (S1).

Next, the preprocessing unit 12 performs preprocessing on the two-dimensional image acquired by the image acquisition unit 11 and outputs the preprocessed image to the first area extraction unit 13 and the image identification unit 14. (S2).

Next, the first region extracting unit 13 extracts a body hair region from the input image that has been preprocessed by the preprocessing unit 12 (S3). Then, the image identifying unit 14 identifies the body hair region extracted by the first region extracting unit 13 from the input image that has been preprocessed by the preprocessing unit 12 (S4).

Finally, the second area extraction unit 15 extracts the microvessel area from the image after the body hair area has been identified by the image identification unit 14 (S5).

<Effects of image processing device 1>
As described above, according to the image processing device 1 according to the present embodiment, the second region extraction section 15 extracts the microvessel region from the image after the body hair region has been identified by the image identification section 14. do. Therefore, the second region extraction unit 15 can extract microvessels with high precision.

Furthermore, since the image processing device 1 extracts microvessels using a two-dimensional image, it is possible to realize an image processing system including the image capturing device 2 at low cost.

(Embodiment 2)
The image processing device 1 according to the first embodiment extracts a microvessel region from an image after a body hair region has been identified. The image processing device 1A according to the second embodiment separately extracts a body hair region and a microvessel region, and identifies the region extracted as body hair from the region extracted as microvessels.

FIG. 10 is a diagram showing a configuration example of an image processing apparatus 1A according to Embodiment 2 of the present invention. The image processing device 1A includes an image acquisition section 11, a preprocessing section 12, a first region extraction section 13, an image identification section 14A, and a second region extraction section 15A. Note that components having the same functions as those described in Embodiment 1 are given the same reference numerals, and the description thereof will be omitted as appropriate.

The image acquisition unit 11 acquires an image including microvessels from the image capturing device 2 via the communication network 3. The image acquisition unit 11 outputs the image acquired from the image capturing device 2 to the preprocessing unit 12.

The preprocessing unit 12 performs normalization processing to equalize brightness values and adjust brightness and darkness in all images. Then, the preprocessing unit 12 performs binarization processing on the image after performing the normalization processing to extract a region having a brightness equal to or less than a threshold value.

The first area extraction unit 13 extracts a body hair area from the image that has been preprocessed by the preprocessing unit 12 . The first region extraction unit 13 is trained to extract the body hair region using U-Net, CNN, etc., as described above.

The second region extracting section 15A extracts a region of microvessels from the image that has been preprocessed by the preprocessing section 12. The second region extraction unit 15A is trained to extract microvessel regions using U-Net, CNN, etc., as described above.

The image identification section 14A identifies the image of the body hair region extracted by the first region extraction section 13 from the image of the microvessel region extracted by the second region extraction section 15A. This process is performed in order to identify body hair that has been erroneously determined to be a microvessel by the image identification unit 14A when the second region extraction unit 15A has erroneously determined the body hair to be a microvessel.

For example, the image identification section 14A may remove the image of the body hair region extracted by the first region extraction section 13 from the image of the microvessel region extracted by the second region extraction section 15A. The image identification unit 14A also changes the image so that the image of the body hair area extracted by the first area extraction unit 13 can be determined from the image of the microvessel area extracted by the second area extraction unit 15A. You can do it like this.

<Processing procedure of image processing device 1A>
FIG. 11 is a flowchart for explaining the processing procedure of the image processing apparatus 1A according to the second embodiment of the present invention. First, the image acquisition unit 11 acquires a two-dimensional image photographed by the image photographing device 2 (S11).

Next, the preprocessing unit 12 performs preprocessing on the two-dimensional image acquired by the image acquisition unit 11, and the preprocessed image is transferred to the first area extraction unit 13 and the second area extraction unit 15A. (S12).

Next, the first region extracting section 13 extracts a body hair region from the input image that has been preprocessed by the preprocessing section 12 (S13). Furthermore, the second region extracting section 15A extracts a region of microvessels from the input image that has been preprocessed by the preprocessing section 12 (S14).

Finally, the image identifying section 14A identifies the image of the body hair region extracted by the first region extracting section 13 from the image of the microvessel region extracted by the second region extracting section 15A (S15).

<Effects of image processing device 1A>
As explained above, according to the image processing device 1A according to the present embodiment, the image identification section 14A uses the first region extraction section 13 from the image of the microvessel region extracted by the second region extraction section 15A. Identify the image of the extracted body hair region. Therefore, the image identification unit 14A can extract microvessels with high precision.

Moreover, since the image processing device 1A extracts microvessels using a two-dimensional image, it is possible to realize an image processing system including the image capturing device 2 at low cost.

(Embodiment 3)
The image processing device 1 according to the first embodiment and the image processing device 1A according to the second embodiment extract microvessels and body hair using machine learning such as U-Net and CNN. The image processing device 1B according to the third embodiment extracts the tips of snare portions by image processing and determines microvessels.

FIG. 12 is a diagram showing a configuration example of an image processing apparatus 1B according to Embodiment 3 of the present invention. The image processing device 1B includes an image acquisition section 11, a preprocessing section 12B, a region extraction section 16, and an image identification section 14B. Note that components having the same functions as those described in Embodiments 1 and 2 are designated by the same reference numerals, and the description thereof will be omitted as appropriate.

The image acquisition unit 11 acquires an image including microvessels from the image capturing device 2 via the communication network 3. The image acquisition unit 11 outputs the image acquired from the image capturing device 2 to the preprocessing unit 12.

The preprocessing unit 12B performs normalization processing to equalize brightness values and adjust brightness and darkness in all images. Note that in this embodiment, the binarization process is not performed by the preprocessing unit 12B but by the area extraction unit 16.

The area extraction unit 16 determines microvessels from the image preprocessed by the preprocessing unit 12B. Then, the image identifying section 14B identifies the image of the region determined to be a microvessel by the region extracting section 16, and outputs the identified image.

For example, the image identification unit 14B may remove images (objects) other than the areas determined to be microvessels by the area extraction unit 16. Further, the image identifying unit 14B may change the image so that it can identify the image of the area determined by the area extracting unit 16 to be a microvessel. For example, the image identification unit 14B can differentiate microvessel areas and microvessel areas by changing the color or gradation of each area in an image obtained by binarizing the image preprocessed by the preprocessing unit 12B. It may also be possible to make it possible to distinguish between other areas (including areas of body hair).

FIG. 13 is a diagram showing an example of the configuration of the area extraction section 16. The region extraction section 16 includes a snare tip extraction section 161, a candidate region extraction section 162, and a microvessel determination section 163.

The snare tip extraction section 161 extracts the snare tip from the image preprocessed by the preprocessing section 12B by image processing. The snare tip extraction unit 161 first detects objects from the image preprocessed by the preprocessing unit 12B, and obtains coordinate information of each object. Then, the snare tip extracting unit 161 extracts a black round object from each object as the snare tip, and obtains coordinate information of the snare tip.

Specifically, the snare tip extracting unit 161 performs image processing on an image obtained by binarizing a two-dimensional image using a first threshold value (an image in which microvessels and body hair have disappeared and the snare tip remains). An object (a dotted circular area) corresponding to the tip of the snare is extracted. Note that any image processing may be used as long as it can extract a circular region.

The candidate region extracting section 162 extracts a binary image (an image in which microvessels and body hair remain) obtained by binarizing the two-dimensional image using a second threshold different from the first threshold, in the same way as the first region extracting section 13 of the first embodiment. Then, extract the body hair region. The candidate region extraction unit 162 then removes the body hair region from the binarized image.

Then, the candidate region extraction unit 162 extracts a plurality of candidate regions of microvessels as objects from the image from which body hair regions have been removed, and obtains coordinate information of each object. The candidate region extraction unit 162 has been machine-learned in advance to extract microvessels.

The microvessel determination unit 163 refers to the coordinate information of each object and identifies, among the microvessel candidate regions, a candidate region that includes (overlaps) the object at the snare tip as a microvessel region.

Note that the second region extracting section 15 described in the first embodiment or the second region extracting section 15A described in the second embodiment has a configuration similar to that of the region extracting section 16, and The tip of the hoof may be extracted, and microvessels may be determined based on the extracted tip of the snare.

<Processing procedure of image processing device 1B>
FIG. 14 is a flowchart for explaining the processing procedure of the image processing apparatus 1B according to the third embodiment of the present invention. First, the image acquisition unit 11 acquires a two-dimensional image photographed by the image photographing device 2 (S21).

Next, the preprocessing unit 12B performs preprocessing on the two-dimensional image acquired by the image acquisition unit 11, and outputs the preprocessed image to the area extraction unit 16 (S22).

Next, the snare tip extraction unit 161 extracts the object of the snare tip from the input image that has been preprocessed by the preprocessing unit 12B (S23). After removing the body hair region from the two-dimensional image, the candidate region extraction unit 162 extracts a plurality of candidate regions of microvessels (S24).

The microvessel determination unit 163 determines, among the microvessel candidate regions, a candidate region that includes (overlaps) the object at the snare tip as a microvessel region (S25). Finally, the image identifying unit 14B identifies the image of the region determined to be a microvessel by the microvessel determining unit 163 (S26).

FIG. 15 is a diagram showing regions determined to be body hair by the region extraction unit 16. In FIG. 15, the shaded area indicates the area determined as body hair. The black area indicates the area determined to be a part other than body hair.

FIG. 16 is a diagram showing an image after microvessels have been identified by the image identifying unit 14B. In FIG. 16, the shaded areas indicate areas determined as microvessels. As shown in FIG. 16, regions determined to be microvessels can be distinguished from other regions containing body hair.

<Effects of image processing device 1B>
As described above, according to the image processing device 1B according to the present embodiment, the area extraction unit 16 removes the body hair area from the image preprocessed by the preprocessing unit 12B, and then Determine microvessels based on Then, the image identifying unit 14B identifies the image of the area determined to be a microvessel by the area extracting unit 16. Therefore, the image identification unit 14B can extract microvessels with high precision.

Furthermore, since the image processing device 1B extracts microvessels using a two-dimensional image, it is possible to realize an image processing system including the image capturing device 2 at low cost.

<Example of implementation using software>
The control blocks of the image processing devices 1, 1A, and 1B (in particular, the preprocessing section 12, the first region extraction section 13, the image identification section 14, 14A, 14B, the second region extraction section 15, 15A, and the region extraction section 16) are as follows. , may be realized by a logic circuit (hardware) formed on an integrated circuit (IC chip) or the like, or may be realized by software.

In the latter case, the image processing devices 1, 1A, and 1B are equipped with a computer that executes instructions of a program that is software that implements each function. The computer includes, for example, at least one processor and at least one computer-readable recording medium storing the program. In the computer, the processor reads the program from the recording medium and executes the program, thereby achieving the object of the present invention. As the processor, for example, a CPU (Central Processing Unit) can be used. As the recording medium, in addition to "non-temporary tangible media" such as ROM (Read Only Memory), tapes, disks, cards, semiconductor memories, programmable logic circuits, etc. can be used. Further, the computer may further include a RAM (Random Access Memory) for expanding the above program. Furthermore, the program may be supplied to the computer via any transmission medium (communication network, broadcast waves, etc.) that can transmit the program. Note that one aspect of the present invention can also be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission.

〔summary〕
The image processing device according to aspect 1 of the present invention includes:
a first region extraction unit that extracts a body hair region from the two-dimensional image;
an image identification unit that identifies the body hair region extracted by the first region extraction unit from the two-dimensional image;
a second region extraction section that extracts a venule or arteriole region from the image in which the body hair region has been identified by the image identification section.

According to the above configuration, the second region extraction section extracts the venule or arteriole region from the image after the body hair region has been identified by the image identification section, so the venule or arteriole can be extracted with high precision. can be extracted.

The image processing device according to aspect 2 of the present invention includes:
a first region extraction unit that extracts a body hair region from the two-dimensional image;
a second region extraction unit that extracts a region of a venule or an arteriole from the two-dimensional image;
Identifying a region of the venule or arteriole that is not the body hair region extracted by the first region extraction section from an image of the venule or arteriole region extracted by the second region extraction section. and an image identification unit.

According to the above configuration, the image identification section identifies the image of the body hair region extracted by the first region extraction section from the image of the venule or arteriole region extracted by the second region extraction section. Venules or arterioles can be extracted with high precision.

The image processing device according to aspect 3 of the present invention is the image processing device according to aspect 1 or 2, comprising:
The second region extracting unit extracts a tip of the snare portion, and determines a region of the venule or arteriole based on the tip of the snare portion.

According to the above configuration, the second region extraction section can extract venules or arterioles with even higher precision.

The image processing device according to aspect 4 of the present invention is the image processing device according to aspect 1 or 2,
The first region extraction unit is trained by machine learning to extract the body hair region from the two-dimensional image,
The second region extracting unit is trained by machine learning to extract the region of the venule or the arteriole from the two-dimensional image.

According to the above configuration, by increasing the learning accuracy of the first region extracting section and the second region extracting section, venules or arterioles can be extracted with higher precision.

The image processing device according to aspect 5 of the present invention includes:
a snare tip extraction unit that extracts a snare tip from a two-dimensional image;
a candidate region extraction unit that extracts a candidate region of a venule or an arteriole from the two-dimensional image;
A microvessel determination unit that determines a region of the venule or arteriole based on the snare tip and the candidate region extracted by the snare tip extraction unit.

According to the above configuration, the microvessel determination section determines the region of the venule or arteriole based on the snare tip and candidate region extracted by the snare tip extraction section, so Arteries can be extracted with high precision.

An image processing device according to aspect 6 of the present invention is the image processing device according to aspect 5, comprising:
The microvessel determination section determines the candidate region including the snare tip extracted by the snare tip extraction section to be the venule or arteriole region.

According to the above configuration, capillaries can be extracted with higher precision.

The image processing method according to aspect 7 of the present invention includes:
Extract the body hair region from the two-dimensional image,
identifying the extracted body hair region from the two-dimensional image;
A venule or arteriole region is extracted from the image in which the body hair region has been identified.

According to the above configuration, since the venule or arteriole region is extracted from the image after the body hair region has been identified, the venule or arteriole can be extracted with high precision.

The image processing method according to aspect 8 of the present invention includes:
Extract the body hair region from the two-dimensional image,
extracting a region of venules or arterioles from the two-dimensional image;
A region of the venule or arteriole that is not the extracted body hair region is identified from an image of the extracted region of the venule or arteriole.

According to the above configuration, since the image of the extracted body hair region is identified from the image of the extracted venule or arteriole region, the venule or arteriole can be extracted with high precision.

The image processing method according to aspect 9 of the present invention includes:
Extract the snare tip from the two-dimensional image,
extracting candidate regions of venules or arterioles from the two-dimensional image;
The region of the venule or arteriole is determined based on the extracted tip of the snare portion and the candidate region.

According to the above configuration, since the region of the venule or arteriole is determined based on the extracted tip of the snare portion and the candidate region, the venule or arteriole can be extracted with high precision.

[Additional notes]
The present invention is not limited to the embodiments described above, and various modifications can be made within the scope of the claims, and embodiments obtained by appropriately combining technical means disclosed in different embodiments. are also included within the technical scope of the present invention.

1, 1A, 1B Image processing device 2 Image photographing device 3 Communication network 11 Image acquisition section 12, 12B Preprocessing section 13 First region extraction section 14, 14A, 14B Image identification section 15, 15A Second region extraction section 16 Region extraction Part 21 Camera 22 Green light source 23 Slot 24 Blood collection needle 25 Hydrogel 100 Image processing system 161 Snare tip extraction unit 162 Candidate region extraction unit 163 Microvessel determination unit

Claims (8)

a first region extraction unit that extracts a body hair region from the two-dimensional image;
an image identification unit that identifies the body hair region extracted by the first region extraction unit from the two-dimensional image;
a second area extraction unit that extracts a venule or arteriole area from the image in which the body hair area has been identified by the image identification unit;
The second region extracting unit is an image processing device that extracts a tip of a snare portion and determines a region of the venule or the arteriole based on the tip of the snare portion. a first region extraction unit that extracts a body hair region from the two-dimensional image;
A second region for extracting a tip of a snare from the two-dimensional image, determining a region of a venule or an arteriole based on the tip of the snare, and extracting a region of the venule or arteriole. an extraction section;
Identifying a region of the venule or arteriole that is not the body hair region extracted by the first region extraction section from an image of the venule or arteriole region extracted by the second region extraction section. An image processing device comprising: an image identification unit that performs The first region extraction unit is trained by machine learning to extract the body hair region from the two-dimensional image,
The image processing device according to claim 1 or 2, wherein the second region extraction unit is trained to extract the region of the venule or the arteriole from the two-dimensional image by machine learning. a snare tip extraction unit that extracts a snare tip from a two-dimensional image;
a candidate region extraction unit that extracts a candidate region of a venule or an arteriole from the two-dimensional image;
An image processing device comprising: a microvessel determining unit that determines a region of the venule or arteriole based on the snare tip and the candidate region extracted by the snare tip extracting unit. The image processing device according to claim 4 , wherein the microvessel determination unit determines the candidate region including the snare tip extracted by the snare tip extraction unit to be a region of the venule or the arteriole. . Extract the body hair region from the two-dimensional image,
identifying the extracted body hair region from the two-dimensional image;
extracting a venule or arteriole region from the image in which the body hair region has been identified;
An image processing method, wherein in the step of extracting a region of the venule or the arteriole, a tip of the snare portion is extracted, and a region of the venule or the arteriole is determined based on the tip of the snare portion. Extract the body hair region from the two-dimensional image,
extracting a tip of the snare from the two-dimensional image, determining a region of a venule or arteriole based on the tip of the snare, and extracting a region of the venule or arteriole;
An image processing method for identifying a region of the venule or arteriole that is not the extracted body hair region from an image of the extracted region of the venule or arteriole. Extract the snare tip from the two-dimensional image,
extracting candidate regions of venules or arterioles from the two-dimensional image;
An image processing method, wherein a region of the venule or the arteriole is determined based on the extracted tip of the snare portion and the candidate region.

Images