JP6931425B2 - Medical image learning device, medical image learning method, and program - Google Patents

Description

The present invention relates to a medical image learning device, a medical image learning method, and a program, and particularly relates to learning of a special light observation image.

In the medical field, AI technology applying deep learning called deep learning is expected. Examples of AI technology include automatic detection of lesions and automatic discrimination of lesions. AI is an abbreviation for Artificial Intelligence.

Patent Document 1 describes an electronic endoscopy device that captures a wavelength region of visible light to generate a normal image that is a color image representing the visible light region and a diagnostic image that is a narrow-band spectroscopic image. .. The document describes that an arithmetic process is applied to a normal image to acquire an image equivalent to a narrowband image obtained by using a narrowband bandpass filter to generate a diagnostic image. ..

Patent Document 2 describes a machine learning support device that supports machine learning of an artificial intelligence device and verification of operation of the artificial intelligence device for multi-sensor information. The document describes that virtual data for learning is generated by randomly changing other waveform parts while retaining the characteristic spectrum part of the spectroscopic data. Further, the document describes that an artificial intelligence device is trained in advance using a virtual data set.

Japanese Unexamined Patent Publication No. 2010-75368 JP-A-2017-102755

However, the development of a learning model requires a large amount of correct data sets. The correct data set referred to here means an image and a set of image recognition results. In the medical field, collecting correct data sets is a barrier to the development of learning models. Observation using an endoscope device includes not only observation using normal light but also observation using special light such as BLI (Blue LAZER Imaging). The development of AI that can handle multiple modes will also require the collection of large correct data sets.

White light can be mentioned as an example of the normal light referred to here. White light can include light in multiple wavelength bands. Further, as an example of special light, light in a band narrower than the wavelength band of white light can be mentioned. The special light may include light in a plurality of wavelength bands.

Patent Document 1 does not describe learning. Further, although Patent Document 1 has a description such as adjustment of brightness, there is no description regarding switching of processing focusing on the difference in band components in the spectral distribution.

Patent Document 2 describes a virtual data set for learning generated by randomly changing other wavelength components while retaining the spectral components of the spectroscopic data. On the other hand, Patent Document 2 does not describe that the band components in the corresponding bands are used for learning in a plurality of image groups having different spectral distributions.

The present invention has been made in view of such circumstances, and can realize learning corresponding to a plurality of images having different spectral distributions without collecting a large number of correct images for each image having different spectral distributions. It is an object of the present invention to provide an image learning device, a medical image learning method, and a program.

In order to achieve the above object, the following aspects of the invention are provided.

The medical image learning device according to the first aspect includes an image acquisition unit that acquires a first image and a second image having different spectral distributions from each other, and an image processing unit that performs image processing on the first image to generate a third image. , A first image group containing a plurality of first images, a second image group containing a plurality of second images, a third image group containing a plurality of third images, and a first image group, a second image group, and A learning unit for learning a recognizer to be applied to automatic recognition using each correct answer data of the third image group is provided, and the image processing unit is a second image in the band included in the spectral distribution of the first image. Image processing that suppresses signals in bands that have different characteristics from the corresponding bands, and emphasizes signals in bands that have the same characteristics as the corresponding bands in the second image or signals in similar bands among the bands included in the first image. This is a medical image learning device that generates a third image from a first image by performing at least one of the image processing to be performed.

According to the first aspect, for the first image and the second image having different spectral distributions, among the bands included in the spectral distribution of the first image, the signal of the band having the same characteristics as the corresponding band of the second image or A third image is generated by emphasizing signals in a similar band or suppressing signals in a band whose characteristics differ from those of the corresponding band in the second image. Automatic recognition using the first image group including a plurality of first images, the second image group including a plurality of second images, the third image group including a plurality of third images, and the correct answer data of each image group. Learn the classifiers that apply to. As a result, learning for each image group having a different spectral distribution can be realized without collecting a large amount of correct images for each image group having a different spectral distribution.

An example of an image having a different spectral distribution is an image to which different observation modes are applied. As an example of the observation mode, there is a mode in which the illumination light is different.

In the second aspect, in the medical image learning device of the first aspect, the image processing unit may be configured to perform a process of converting the luminance dispersion on the first image.

According to the second aspect, a process of converting the luminance variance to the first image can be applied to generate a third image from the first image.

In the third aspect, in the medical image learning device of the first aspect, the image processing unit may be configured to perform a process of changing the brightness of the first image to a predetermined brightness value.

According to the third aspect, a process of changing the brightness to a predetermined brightness value can be applied to the first image to generate a third image from the first image.

In the fourth aspect, in the medical image learning device of the first aspect, the image processing unit may be configured to perform a process of changing the brightness of the first image to a random brightness value.

According to the fourth aspect, a process of changing the brightness to a random brightness value can be applied to the first image to generate a third image from the first image.

In the fifth aspect, in the medical image learning device of the first aspect, the first image is a normal light image captured by using normal light, and the second image is captured by using special light having a narrower band than the normal light. It may be configured as a special light image.

According to the fifth aspect, the normal light observation mode can be applied to generate the first image. In addition, a special light observation mode can be applied to generate a second image.

A sixth aspect is the medical image learning apparatus of the fifth aspect, in which the image processing unit may be configured to perform a process of suppressing a B channel component on a normal optical image.

According to the sixth aspect, the B channel component of the normal optical image can be suppressed to generate a third image.

In the seventh aspect, in the medical image learning device of the fifth aspect, the image processing unit may be configured to perform a process of emphasizing at least one of an R channel component and a G channel component on a normal optical image.

According to the seventh aspect, at least one of the R channel component and the G channel component of the normal optical image can be emphasized to generate a third image.

In the eighth aspect, in the medical image learning device of the fifth aspect, the image processing unit may be configured to perform a process of suppressing an R channel component on a normal optical image.

According to the eighth aspect, the R channel component of the normal optical image can be suppressed to generate a third image.

As an example of the special optical image in the eighth aspect, there is an image in which the R channel component of the normal optical image is missing.

In the ninth aspect, in the medical image learning device of the fifth aspect, the image processing unit may be configured to perform a process of emphasizing the B channel component and the G channel component on the normal optical image.

According to the ninth aspect, the B channel component and the G channel component of the normal optical image can be emphasized to generate a third image.

As an example of the special optical image in the ninth aspect, there is an image in which the R channel component of the normal optical image is missing.

A tenth aspect is the medical image learning apparatus of the first aspect, in which the image processing unit suppresses a predetermined frequency component of the spatial frequency in the processing target band of the first image, or the space in the processing target band of the first image. It may be configured to perform a process of emphasizing a specified frequency component of the frequency.

According to the tenth aspect, among the spatial frequencies of the first image, the frequency components different from those of the second image are suppressed, or the frequency components that are the same as or similar to those of the second image are emphasized, from the first image to the third. Images can be generated.

In the eleventh aspect, in the medical image learning apparatus of the tenth aspect, the first image is a normal light image captured by using normal light, and the second image is captured by using special light having a narrower band than the normal light. It is a special optical image, and the image processing unit may be configured to perform processing for emphasizing the high frequency component of the B channel component of the normal optical image.

According to the eleventh aspect, the high frequency component of the spatial frequency in the B channel component of the normal light image can be emphasized to generate a third image from the normal light image.

A twelfth aspect is the medical image learning device according to any one of the first to eleventh aspects, wherein the learning unit switches between learning methods of the first image group and learning of the second image group. good.

According to the twelfth aspect, the accuracy of learning can be improved.

The thirteenth aspect is the medical image learning device according to any one of the first to eleventh aspects, wherein the learning unit switches the learning method between the learning of the second image group and the learning of the third image group. May be.

According to the thirteenth aspect, the accuracy of learning can be improved.

A fourteenth aspect may be a configuration in which the medical image learning device of the twelfth aspect or the thirteenth aspect applies a learning method for suppressing or emphasizing the influence of the third image on learning.

According to the fourteenth aspect, the influence of the third image on learning can be suppressed or emphasized. This can improve the accuracy of learning.

The fifteenth aspect is the medical image learning device of the twelfth aspect or the thirteenth aspect, and the learning unit receives the correct answer data of the first image group, the third image group, the first image group, and the correct answer data of the third image group. For the trained recognizer used or the trained learner using the correct answer data of the third image group and the third image group, learning using the correct answer data of the second image group and the second image group is performed. It may be a configuration to be implemented.

According to the fifteenth aspect, the correct answers of the second image group and the second image group, which are true data with higher reliability than the third image group, for at least the trained recognizer using the third image group. Perform re-learning using the data. As a result, the accuracy of learning can be improved, and the accuracy of the recognizer can be improved.

The medical image learning method according to the 16th aspect includes an image acquisition step of acquiring a first image and a second image having different spectral distributions from each other, and an image processing step of performing image processing on the first image to generate a third image. , A first image group containing a plurality of first images, a second image group containing a plurality of second images, a third image group containing a plurality of third images, and a first image group, a second image group, and The image processing step includes a learning step of learning a recognizer to be applied to automatic recognition using each correct answer data of the third image group, and the image processing step is a second image in the band included in the spectral distribution of the first image. Image processing that suppresses signals in bands that have different characteristics from the corresponding bands, and emphasizes signals in bands that have the same characteristics as the corresponding bands in the second image or signals in similar bands among the bands included in the first image. This is a medical image learning method for generating a third image from a first image by performing at least one of the image processing to be performed.

According to the 16th aspect, the same effect as that of the 1st aspect can be obtained.

In the 16th aspect, the same items as those specified in the 2nd to 15th aspects can be appropriately combined. In that case, the component responsible for the process or function specified in the medical image learning device can be grasped as the component of the medical image learning method responsible for the corresponding process or function.

The program according to the seventeenth aspect includes an image acquisition function for acquiring a first image and a second image having different spectral distributions from each other, an image processing function for performing image processing on the first image to generate a third image, and an image processing function. A first image group including a plurality of first images, a second image group including a plurality of second images, a third image group including a plurality of third images, and a first image group, a second image group, and a first image group. It is a program that realizes a learning function that trains a recognizer to be applied to automatic recognition using the correct answer data of each of the three image groups, and the image processing function is the third of the bands included in the spectral distribution of the first image. (2) Image processing that suppresses signals in bands with characteristics different from the corresponding bands of the image, and signals in the same band or similar bands as the corresponding band in the second image among the bands included in the first image. It is a program that generates a third image from a first image by performing at least one of image processing that emphasizes.

According to the 17th aspect, the same effect as that of the 1st aspect can be obtained.

In the 17th aspect, the same items as those specified in the 2nd to 15th aspects can be appropriately combined. In that case, the component responsible for the process or function specified in the medical image learning device can be grasped as the component of the program responsible for the corresponding process or function.

According to the present invention, for the first image and the second image having different spectral distributions, among the bands included in the spectral distribution of the first image, signals or similar bands having the same characteristics as the corresponding band of the second image. The signal of the band to be used is emphasized, or the signal of the band having different characteristics from the corresponding band of the second image is suppressed to generate the third image. Automatic recognition using the first image group including a plurality of first images, the second image group including a plurality of second images, the third image group including a plurality of third images, and the correct answer data of each image group. Learn the classifiers that apply to. As a result, learning for each image group having a different spectral distribution can be realized without collecting a large amount of correct images for each image group having a different spectral distribution.

FIG. 1 is a block diagram showing an example of the hardware configuration of the endoscopic image learning device according to the embodiment. FIG. 2 is a functional block diagram of the endoscopic image learning device according to the embodiment. FIG. 3 is a functional block diagram of the learning unit shown in FIG. FIG. 4 is a flowchart showing the procedure of the endoscopic image learning method according to the embodiment. FIG. 5 is an explanatory diagram of the spectral sensitivity of the image sensor. FIG. 6 is an explanatory diagram showing the relationship between the intensity distribution of the illumination light and the spectral distribution of the endoscopic image. FIG. 7 is an explanatory diagram showing the relationship between the spectral distribution of the normal optical image and the spectral distribution of the special optical image in NBI. FIG. 8 is an overall configuration diagram of the endoscope system. FIG. 9 is a functional block diagram of the endoscopic system. FIG. 10 is a graph showing the light intensity distribution.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the present specification, the same components are designated by the same reference numerals, and duplicate description will be omitted as appropriate.

[Endoscopic image learning device]
The endoscopic image learning device shown in the present embodiment expands the number of training data by combining images acquired in different observation modes. By using the images acquired in different observation modes for learning, highly accurate learning can be realized even when the number of images acquired in a specific observation mode is not sufficient. This makes it possible to realize highly accurate AI technology. The endoscopic image learning device shown in the embodiment is an example of a medical image learning device.

In this embodiment, learning applied to an automatic recognition system of a region of interest using LCI (Linked Color Imaging) is illustrated. In addition, LCI means a special light color enhancement function. In the following explanation, the term learning and the term machine learning will be treated as synonymous.

[Hardware configuration of endoscopic image learning device]
FIG. 1 is a block diagram showing an example of the hardware configuration of the endoscopic image learning device according to the embodiment. The endoscopic image learning device 10 may apply a personal computer or a workstation.

The endoscopic image learning device 10 includes a communication unit 12, a first storage device 14, a second storage device 16, a third storage device 17, an operation unit 18, a CPU 20, a RAM 22, a ROM 24, and a display unit 26. CPU is an abbreviation for Central Processing Unit. RAM is an abbreviation for Random Access Memory. ROM is an abbreviation for Read Only Memory.

The communication unit 12 is an interface that processes communication with an external device. The communication unit 12 may apply a general-purpose communication interface. The communication unit 12 can be either wired or wireless.

The first storage device 14 includes a first data set including a normal light image group including a plurality of normal light images captured by applying the normal light observation mode and correct answer data indicating a correct image recognition result of each normal light image. Be remembered.

The second storage device 16 applies a special light observation mode to store a special light image group including a plurality of special light images and a second data set including correct answer data indicating a correct image recognition result of each special light image. ..

The third storage device 17 is a third data set including a processed image group including a plurality of processed images obtained by subjecting a normal light image or a special light image to image processing, and correct answer data indicating a correct image recognition result of each processed image group. Is remembered.

In this embodiment, an example in which a processed image is generated from a normal optical image and used for learning will be described. Correct answer data representing the correct recognition result of the processed image can be extracted from the first dataset. The processed image may be generated from the special light image, and the correct answer data of the processed image may be extracted from the second data set.

The first storage device 14, the second storage device 16, and the third storage device 17 may apply a large-capacity storage device. The first storage device 14, the second storage device 16, and the third storage device 17 may apply a storage device arranged outside the endoscopic image learning device 10. The first storage device 14, the second storage device 16, and the third storage device 17 may be communicably connected to the endoscopic image learning device 10 via a network.

The normal light image shown in the embodiment is an example of the first image. The normal light image group shown in the embodiment is an example of the first image group. The special light image shown in the embodiment is an example of the second image. The special optical image group shown in the embodiment is an example of the second image group.

The normal light image is a color image obtained by irradiating normal light and taking an image in the observation of the inside of the body cavity using an endoscope device. The special light image is a color image obtained by irradiating and imaging the inside of the body cavity using a special light.

Normal light is light in which light in all wavelength bands of visible light is mixed almost evenly. Normal light images are usually used for observation. White light is an example of normal light. A relatively large number of ordinary optical image groups can be collected.

The special light is light in various wavelength bands according to the purpose of observation, which is a combination of light in one specific wavelength band or light in a plurality of specific wavelength bands. The special light has a band narrower than the white wavelength band, and is used for NBI (Narrow band imaging), LCI, FICE (Flexible spectral imaging color enhancement), and the like. NBI means narrow band observation.

A first example of a particular wavelength band is, for example, the blue or green band in the visible range. The wavelength band of the first example includes a wavelength band of 390 nanometers or more and 450 nanometers or less, or 530 nanometers or more and 550 nanometers or less. The light of the first example has a peak wavelength in the wavelength band of 390 nanometers or more and 450 nanometers or less, or 530 nanometers or more and 550 nanometers or less.

A second example of a particular wavelength band is, for example, the red band in the visible range. The wavelength band of the second example includes a wavelength band of 585 nanometers or more and 615 nanometers or less, or 610 nanometers or more and 730 nanometers or less. The light of the second example has a peak wavelength in the wavelength band of 585 nanometers or more and 615 nanometers or less or 610 nanometers or more and 730 nanometers or less.

The third example of a specific wavelength band includes a wavelength band in which the absorption coefficient differs between oxidized hemoglobin and reduced hemoglobin, and the light in the third example has a peak wavelength in a wavelength band in which the absorption coefficient differs between oxidized hemoglobin and reduced hemoglobin. Has. The wavelength band of the third example includes a wavelength band of 400 ± 10 nanometers, 440 ± 10 nanometers, 470 ± 10 nanometers, or 600 nanometers or more and 750 nanometers or less. The light of the third example has a peak wavelength in the wavelength band of 400 ± 10 nanometers, 440 ± 10 nanometers, 470 ± 10 nanometers, or 600 nanometers or more and 750 nanometers or less.

The fourth example of a specific wavelength band is used for fluorescence observation, which is observation of fluorescence emitted by a fluorescent substance in a living body. The wavelength band of the fourth example is the wavelength band of the excitation light that excites the fluorescent substance, for example, the wavelength band from 390 nanometers to 470 nanometers.

The fifth example of a specific wavelength band is the wavelength band of infrared light. The wavelength band of the fifth example includes a wavelength band of 790 nanometers or more and 820 nanometers or less, or 905 nanometers or more and 970 nanometers or less. The light of the fifth example has a peak wavelength in a wavelength band of 790 nanometers or more and 820 nanometers or less, or 905 nanometers or more and 970 nanometers or less.

The special light image captured under special light having such a specific wavelength band is for obtaining an easy-to-see image according to the purpose of observing the lesion, for example, for the purpose of observing the surface structure. It is used only when it corresponds to, and the number of data is not large.

In the present embodiment, the first data set of the normal optical image group stored in the first storage device 14 is prepared more than the second data set of the special optical image group stored in the second storage device 16. It is assumed that A number is prepared for the third data set to make up for the shortage of the second data set.

Further, as an example of correct answer data stored in association with each normal light image, each special light image, and each processed image in the first storage device 14, the second storage device 16, and the third storage device 17, it is usually used. Examples include the type of lesion shown in the optical image and the special optical image, data showing the location of the lesion, and case-specific identification information. As an example of the classification of lesions, there are two classifications, neoplastic or non-neoplastic, or NICE classification. The data indicating the location of the lesion may be rectangular information surrounding the lesion or mask data that covers the lesion. NICE is an abbreviation for NBI International Colorectal Endoscopic Classification.

The first storage device 14, the second storage device 16, and the third storage device 17 include the endoscopic image learning device 10, but the first storage device 14, the second storage device 16, and the third storage device 17 may be arranged outside the endoscopic image learning device 10. In this case, the data set for learning can be acquired from the database outside the endoscope image learning device 10 via the communication unit 12.

As the operation unit 18, a keyboard, a mouse, or the like that is wiredly or wirelessly connected to the computer applied to the endoscopic image learning device 10 is used. The operation unit 18 receives various operation inputs for learning. That is, the operation unit 18 is used as a part of the user interface.

The CPU 20 reads various programs stored in the ROM 24 or a hard disk device (not shown) and executes various processes. As an example of the program to be read, there is an endoscopic image learning program.

The RAM 22 is used as a work area of the CPU 20. Further, the RAM 22 is used as a storage unit for temporarily storing the read program and various data.

As the display unit 26, various monitors such as a liquid crystal monitor that can be connected to a computer applied to the endoscopic image learning device 10 are used. The display unit 26 is used as a part of the user interface. A touch panel type monitor device may be applied to the display unit 26 to integrally configure the display unit 26 and the operation unit 18.

In the endoscope image learning device 10, the CPU 20 reads out the endoscope image learning program stored in the ROM 24, the hard disk device, and the like based on the instruction signal transmitted from the operation unit 18. The CPU 20 executes an endoscopic image learning program.

[Functions of endoscopic image learning device]
<Overview>
FIG. 2 is a functional block diagram of the endoscopic image learning device according to the embodiment. FIG. 2 is a functional block diagram showing the main functions of the endoscopic image learning device 10 shown in FIG. The endoscopic image learning device 10 includes a learning unit 30 and an image generation unit 40.

The learning unit 30 uses the first data set stored in the first storage device 14, the second data set stored in the second storage device 16, and the third data set stored in the third storage device 17. Train and generate a learning model for image recognition. In this embodiment, a convolutional neural network, which is one of the learning models, is constructed. Hereinafter, CNN (Convolution Neural Network) will represent a convolutional neural network.

The image generation unit 40 generates a processed image from a normal optical image. Specifically, with respect to the corresponding wavelength band between the spectral distribution of the normal optical image and the spectral distribution of the special optical image, a process of emphasizing the band component is performed when the band components are the same or similar to each other. When the band components are different from each other, a process of suppressing the band components is performed. The details of the image processing in the image generation unit 40 will be described later. The learning unit 30 and the image generation unit 40 may have a function of an image acquisition unit that acquires a normal light image and a special light image.

<Learning Department>
FIG. 3 is a functional block diagram of the learning unit shown in FIG. The learning unit 30 includes a CNN 32, an error calculation unit 34, and a parameter update unit 36.

The CNN 32 is, for example, a part corresponding to a recognizer that recognizes the type of lesion shown in an endoscopic image. The CNN 32 has a plurality of layer structures and holds a plurality of weight parameters. The CNN 32 can change from an unlearned model to a trained model by updating the weight parameter from the initial value to the optimum value.

The CNN 32 includes an input layer 32a, an intermediate layer 32b, and an output layer 32c. The intermediate layer 32b includes a set 32f composed of a convolution layer 32d and a pooling layer 32e, and a fully connected layer 32g. Each layer of the learning unit 30 has a structure in which a plurality of nodes are connected by using edges.

The normal light image 14a, the special light image 16a, and the processed image 17a to be learned are input to the input layer 32a. The normal light image 14a, the special light image 16a, and the processed image 17a are transmitted to the CNN 32.

The intermediate layer 32b is a portion for extracting features from an image input using the input layer 32a. The convolution layer 32d filters nearby nodes in the previous layer and acquires a feature map. Convolution operation using a filter is applied to the filtering process.

The pooling layer 32e reduces the feature map output from the convolution layer 32d into a new feature map. The convolution layer 32d plays a role of feature extraction such as edge extraction from an image. The pooling layer 32e plays a role of imparting robustness so that the extracted features are not affected by translation or the like.

The intermediate layer 32b is not limited to the case where the convolution layer 32d and the pooling layer 32e are set as one set. For example, the convolutional layer 32d may be continuous or may include a normalized layer (not shown).

The output layer 32c outputs a recognition result for classifying the types of lesions shown in the endoscopic image based on the features extracted using the intermediate layer 32b. The trained CNN32 can, for example, classify endoscopic images into three categories: neoplastic, non-neoplastic, and others. The trained CNN32 can output a score corresponding to neoplasticity, a score corresponding to non-neoplasticity, and a score corresponding to others as recognition results. The total of the three scores is 100%.

Arbitrary initial values are set for the coefficient of the filter applied to each convolution layer 32d of the CNN 32 before learning, the offset value, and the weight of the connection with the next layer in the fully connected layer (not shown).

The error calculation unit 34 acquires the recognition result output from the output layer 32c of the CNN 32, the correct answer data 14b for the normal optical image 14a, and the correct answer data 16b for the special optical image 16a. As the correct answer data corresponding to the processed image 17a, the correct answer data 14b for the normal optical image 14a can be applied.

The error calculation unit 34 calculates the error between the recognition result and the correct answer data. Examples of the error calculation method include softmax cross entropy and sigmoid.

The parameter update unit 36 adjusts the weight parameter of the CNN 32 by applying the error backpropagation method based on the error calculated by the error calculation unit 34. The adjustment process of the weight parameter of CNN32 is repeated, and the learning is repeated until the difference between the output of CNN32 and the correct answer data becomes small.

The learning unit 30 uses all the data sets of the normal optical image group, all the data sets of the special optical image group, and all the data sets of the processed image group, and performs learning to optimize each parameter of the CNN 32. Generate a trained model. The endoscopic image shown in the embodiment is an example of a medical image.

[Procedure of endoscopic image learning method]
FIG. 4 is a flowchart showing the procedure of the endoscopic image learning method according to the embodiment. The endoscopic image learning method shown in FIG. 4 includes a normal optical image acquisition step S10, a special optical image acquisition step S12, an image processing step S14, a processed image storage step S16, a learning step S18, and a recognizer update step S20. .. The endoscopic image learning method shown in the embodiment is an example of a medical image learning method.

In the normal light image acquisition step S10, the learning unit 30 and the image generation unit 40 shown in FIG. 2 read the normal light image 14a from the first storage device 14. In the special light image acquisition step S12, the learning unit 30 reads out the special light image 16a from the second storage device 16. The special optical image acquisition step S12 may be performed by the learning step S18.

In the image processing step S14, the image generation unit 40 performs image processing on the normal optical image 14a to generate a processed image. In the processed image storage step S16, the image generation unit 40 stores the processed image 17a in the third storage device 17.

In the learning step S18, the learning unit 30 reads the processed image 17a from the third storage device 17, and performs learning using the normal light image 14a, the special light image 16a, and the processed image 17a. In the recognizer update step S20, the learning unit 30 updates the recognizer.

[Specific example of image processing using the image generator]
Next, the image processing applied to the image generation unit 40 shown in FIG. 2 will be described in detail. The image generation unit 40 generates a processed image 17a by performing a process of emphasizing or suppressing a signal in the wavelength band in the spectral distribution of the normal optical image 14a.

When the band components in the corresponding wavelength bands between the normal light image 14a and the special light image 16a are different from each other, the image generation unit 40 suppresses the signal in the corresponding wavelength band in the normal light image 14a. do.

On the other hand, when the band components in the corresponding wavelength bands between the normal light image 14a and the special light image 16a are the same or similar, the image generation unit 40 indicates the corresponding wavelength band in the normal light image 14a. The signal of may be emphasized. This makes it possible to increase the number of learning data applicable to the learning of the special light image 16a.

The image generation unit 40 can perform post-processing such as correction to which the reverse processing is applied to the image difference caused by the difference in processing between the observation modes, which is different from the wavelength dependence. The learning unit 30 can apply the corrected image to learning. Examples of processing between observation modes include a linear matrix and a look-up table.

<First Embodiment>
FIG. 5 is an explanatory diagram of the spectral sensitivity of the image sensor. The image sensor is provided in the endoscope that acquires the endoscope image. The image sensor is illustrated with reference numeral 328 in FIG. The endoscope is illustrated with reference numeral 302.

The horizontal axis of the spectral sensitivity characteristic 100 shown in FIG. 5 represents a wavelength. The vertical axis of the spectral sensitivity characteristic 100 represents the sensitivity. The spectral sensitivity characteristic 100 has a B channel component 102 in the wavelength band of the B channel. The spectral sensitivity characteristic 100 has a G channel component 104 in the wavelength band of the G channel. The spectral sensitivity characteristic 100 has an R channel component 106 in the wavelength band of the R channel. In addition, B represents blue. G represents green. R represents red.

FIG. 6 is an explanatory diagram showing the relationship between the intensity distribution of the illumination light and the spectral distribution of the endoscopic image. Reference numeral 120 shown in FIG. 6 represents a normal light intensity distribution. Reference numeral 160 represents the intensity distribution of the illumination light in LCI. The horizontal axis of the intensity distribution 120 of normal light and the intensity distribution 160 of illumination light in LCI represents a wavelength. The vertical axis represents the intensity of the illumination light. The illumination light in LCI is an example of special light.

Reference numeral 140 indicates the spectral distribution of the normal optical image 14a. The horizontal axis of the spectral distribution 140 of the normal optical image 14a represents the wavelength. The vertical axis represents the signal strength.

The spectral distribution 140 of the normal optical image 14a has a B channel component 142 in the wavelength band of the B channel. The spectral distribution 140 of the normal optical image 14a has a G channel component 144 in the wavelength band of the G channel. The spectral distribution 140 of the normal optical image 14a has an R channel component 146 in the wavelength band of the R channel.

Reference numeral 180 represents the spectral distribution of the special optical image 16a in LCI. The horizontal axis of the spectral distribution 180 of the special optical image 16a in LCI represents the wavelength. The vertical axis represents the signal strength. The spectral distribution 180 of the special optical image 16a in the LCI has a B channel component 182 in the wavelength band of the B channel. The spectral distribution 180 of the special optical image 16a in the LCI has a G channel component 184 in the wavelength band of the G channel. The spectral distribution 180 of the special optical image 16a in the LCI has an R channel component 186 in the wavelength band of the R channel.

The spectral distribution 140 of the normal optical image 14a and the spectral distribution 180 of the special optical image 16a in the LCI differ in the characteristics of the B channel component. That is, the B-channel component of the normal optical image 14a and the B-channel component of the special optical image 16a in the LCI both have two peaks, but the magnitudes of the sensitivities of the two peaks are different. In other words, the B channel components of the normal optical image 14a and the special optical image 16a in the LCI are different from each other.

On the other hand, the G channel component and the R channel component in both are equal except for the difference in absolute value. That is, the difference between the normal optical image 14a and the special optical image 16a in the LCI is smaller in the image containing only the G channel component and the R channel component than in the image containing only the B channel component. In other words, the characteristics of the G-channel component and the R-channel component are similar between the normal light image 14a and the special light image 16a in the LCI.

Due to these properties, the image information of the G channel component in the normal light image 14a and the image information of the R channel component in the normal light image 14a are also useful in learning the special light image 16a in the LCI. On the other hand, the image information of the B channel component in the normal light image 14a is considered to be unnecessary or adversely affect the learning of the special light image 16a in the LCI.

Therefore, the image generation unit 40 generates the processed image 17a by performing a process of suppressing the image information of the B channel component in the normal optical image 14a. The learning unit 30 adds the processed image 17a to the normal light image 14a and the special light image 16a to learn the special light image 16a in the LCI.

As an example of the process of suppressing the image information of the B channel component in the normal light image 14a, there is a process of multiplying the brightness value in the normal light image 14a by 1 / α. In addition, α is an arbitrary constant exceeding 1. As another example of the process of suppressing the image information of the B channel component in the normal optical image 14a, there is a process of setting the luminance value to a specified value. As an example of the specified value, there is a brightness value that minimizes the brightness. As the process of suppressing the image information of the B channel component, a process of randomly giving a brightness value of each pixel may be applied.

The brightness value in the normal light image 14a may be shifted in combination with the process of multiplying the brightness value in the normal light image 14a by 1 / α. The process of multiplying the luminance value shown in the embodiment by 1 / α is an example of the process of converting the luminance variance. The process of converting the luminance dispersion may include a process of shifting the luminance value in the normal optical image 14a.

Instead of the process of suppressing the image information of the B channel component in the normal light image 14a, at least one of the image information of the G channel component and the image information of the R channel component in the normal light image 14a is emphasized, and the B channel is relatively relative. The influence of the image information of the component may be reduced.

As an example of the process of emphasizing the image information of the G channel component and the image information of the R channel component in the normal light image 14a, there is a process of multiplying the brightness value of the normal light image 14a by β. Note that β is an arbitrary constant exceeding 1.

The brightness value in the normal light image 14a may be shifted in combination with the process of multiplying the brightness value in the normal light image 14a by β. The process of multiplying the luminance value by β according to the embodiment is an example of the process of converting the luminance variance. The process of converting the luminance dispersion may include a process of shifting the luminance value in the normal optical image 14a.

<Action effect>
According to the image processing according to the first embodiment, the normal optical image 14a is subjected to a treatment for suppressing the B channel component in the spectral distribution or a treatment for emphasizing at least one of the G channel component and the R channel component. This makes it possible to expand the learning data applicable to the learning of the special optical image 16a in the LCI.

<Second Embodiment>
Similar to the image processing according to the first embodiment, the image processing according to the second embodiment is applied to the case where the normal light image 14a is used for learning the special light image 16a in the LCI. In the second embodiment, image processing is performed focusing on the difference between the B channel components in the normal light image 14a and the special light image 16a.

The B-channel component of the special optical image 16a in the LCI contains a large amount of short wavelength components as compared with the B-channel component of the normal optical image 14a. As a result, the special optical image 16a in the LCI has a property of expressing a minute blood vessel structure or the like with a higher contrast than the normal optical image 14a.

The minute blood vessel structure and the like correspond to the high frequency component of the spatial frequency of the special optical image 16a in the LCI. Therefore, the image generation unit 40 generates a processed image 17a obtained by subjecting the image of the B channel component of the normal optical image 14a to a process of emphasizing the high frequency component of the spatial frequency. This makes it possible to bring the image of the B channel component of the normal light image 14a closer to the image of the B channel component of the special light image 16a. Therefore, it is possible to expand the learning data of the special optical image 16a in the LCI.

As an example of defining the high frequency component, there is an example of frequency analysis of the special optical image 16a containing a large amount of the high frequency component and defining the frequency or frequency band to be emphasized based on the frequency distribution. For example, the reference frequency can be calculated, and among the spatial frequency components of the image to be processed, the frequency component above the reference frequency can be used as the high frequency component. The reference frequency can be calculated as the lower limit of the range in which a certain ratio or more of the frequency components included in the image to be processed are included with respect to the whole.

As an example of the process of emphasizing the high frequency component of the spatial frequency, there is a mask process using an unsharp mask. As an example of the process of suppressing the high frequency component of the spatial frequency, there is a filter process using a low-pass filter. The high frequency component of the spatial frequency shown in the embodiment is an example of a defined frequency component of the spatial frequency. The image of the B channel component shown in the embodiment is an example of the processing target band.

<Action effect>
According to the image processing according to the second embodiment, the image of the B channel component of the normal optical image 14a is subjected to a process of emphasizing the high frequency component of the spatial frequency. The processed image 17a shows a minute vascular structure and the like with a higher contrast than the normal optical image 14a. This makes it possible to expand the learning data applicable to the learning of the special optical image 16a in the LCI.

<Third Embodiment>
The image processing according to the third embodiment is applied when the special light image 16a in NBI is used in the learning applied to the automatic recognition system of the region of interest using the normal light image 14a.

FIG. 7 is an explanatory diagram showing the relationship between the spectral distribution of the normal optical image and the spectral distribution of the special optical image in NBI. Reference numeral 200 indicates the spectral distribution of the normal optical image 14a. The horizontal axis of the spectral distribution 200 of the normal optical image 14a represents the wavelength. The vertical axis represents the signal strength.

The spectral distribution 200 of the normal optical image 14a has a B channel component 202 in the wavelength band of the B channel. The spectral distribution 200 of the normal optical image 14a has a G-channel component 204 in the wavelength band of the G-channel. The spectral distribution 200 of the normal optical image 14a has an R channel component 206 in the wavelength band of the R channel.

Reference numeral 220 indicates the spectral distribution characteristics of the special optical image 16a in NBI. The horizontal axis of the spectral distribution 220 of the special optical image 16a in NBI represents the wavelength. The vertical axis represents the signal strength. The special optical image 16a in the NBI has a B channel component 222 in the wavelength band of the B channel. The special optical image 16a in NBI has a G channel component 224 in the wavelength band of the G channel.

On the other hand, in the special light image 16a in NBI, the R channel component is lost at the time of light reception. The spectral distribution 220 of the special optical image 16a in NBI shown in FIG. 7 shows the R channel component lost by using the alternate long and short dash line. That is, in the normal light image 14a and the special light image 16a in NBI, the R channel components are different from each other.

On the other hand, the B channel component 222 and the G channel component 224 of the special optical image 16a in the NBI have different bandwidths from the B channel component 202 and the G channel component 204 of the normal optical image 14a, but there are overlapping bands.

In the normal optical image 14a and the special optical image 16a in the NBI, the B channel component and the G channel component have properties similar to those of the R channel component. That is, in the normal optical image 14a and the special optical image 16a in BI, the B channel components and the G channel components are similar to each other.

Therefore, the image generation unit 40 performs a process of suppressing the R channel component 206 in the normal optical image 14a to generate the processed image 17a. The learning unit 30 may apply the processed image to the learning of the special light image 16a in the NBI.

The image generation unit 40 may generate the processed image 17a by performing a process of emphasizing at least one of the B channel component 202 and the G channel component 204 of the normal optical image 14a. The learning unit 30 may apply the processed image 17a to the learning of the special light image 16a in the NBI. The process of suppressing the band component of the normal optical image 14a to be processed and the process of emphasizing the band component to be processed are the same as those in the first embodiment. The description here will be omitted.

<Action effect>
According to the image processing according to the third embodiment, the normal optical image 14a is subjected to the suppression processing of the R channel component or the enhancement processing of at least one of the G channel component and the B channel component. This makes it possible to expand the learning data applicable to the learning of the special optical image 16a in NBI.

<Application example of the third embodiment>
Since the B-channel component 222 or the G-channel component 224 of the special optical image 16a in the NBI has a large band component having a large hemoglobin absorption coefficient as compared with the B-channel component 222 or the G-channel component 224 of the normal optical image 14a, the NBI has a large band component. The special light image 16a has a property of expressing a blood vessel structure or the like with a higher contrast than the normal light image 14a. Therefore, as in the image processing according to the second embodiment, the image generation unit 40 performs a process of emphasizing the high frequency component at the spatial frequency with respect to the image of the B channel component 222 of the normal optical image 14a or the image of the G channel component 224. This can be done to generate the processed image 17a. The processed image 17a shows a minute blood vessel structure and the like with a higher contrast than the normal optical image 14a. This makes it possible to expand the learning data applicable to the learning of the special optical image 16a in NBI.

<Fourth Embodiment>
The image processing according to the fourth embodiment is applied to the case where learning is performed by applying a plurality of image groups acquired by applying different observation modes. The image processing shown in the first to third embodiments is an endoscopic image obtained by applying a pseudo-original observation mode to an endoscopic image acquired by applying an observation mode different from the original one. It is a process that brings it closer to.

Then, if the influence of another observation mode becomes too large, the improvement of learning accuracy may be less than expected. Therefore, the learning method applied to the learning unit 30 is devised to adjust the balance of the plurality of image groups. Specifically, the learning method of learning the normal light image group and learning the special light image group is switched. Switching the learning method between the learning of the normal light image group and the learning of the special light image group includes switching the learning method between the learning of the processed image 17a generated from the normal light image 14a and the learning of the special light image 16a. .. A specific example of switching the learning method will be described below.

<< Introduction of weighting factor >>
Many learning methods, including CNNs, perform learning that minimizes or maximizes a particular objective function. Therefore, the objective function calculated only from the image group to which the original observation mode is applied is L_maim, and the objective function calculated only from the image group to which the observation mode different from the original is applied is L_sub. Learning is performed to minimize or maximize the objective function L expressed by using Equation 1.

L = L_maim + λ × L_sub ... Equation 1
Note that λ in Equation 1 is a design parameter. Any constant less than 1 is applied to the design parameter λ. When the value of the design parameter λ is made smaller, the influence of the second term, which represents the influence of the observation mode different from the original, can be further suppressed with respect to the first term, which represents the influence of the original observation mode in the objective function L.

When learning is performed using the image group of the normal light image 14a and the image group of the special light image 16a in the LCI, it is as follows. Let L_maim be the objective function calculated only from the image group of the special light image 16a. Let L_sub be the objective function calculated only from the image group of the normal optical image 14a. Learning to minimize or maximize the objective function L of Equation 1 can be performed by appropriately setting the design parameter λ.

The endoscopic image learning device 10 shown in FIG. 2 may include a design parameter setting unit that sets the design parameter λ. Further, the endoscopic image learning device 10 may include an objective function storage unit that stores the objective function L. The learning unit 30 can read the objective function L and the design parameter λ and perform learning.

《Transfer learning》
First, CNN is trained using a plurality of image groups having different observation modes. After that, the CNN is retrained using only the image group to which the original observation mode is applied, with the trained parameters as the initial values.

Specifically, the trained CNN using the normal light image 14a and the processed image 17a or the trained CNN using the processed image 17a is relearned by using the special light image 16a.

As a result, it is possible to use the information of the image group to which the observation mode different from the original is applied and to bring the image group closer to the image group to which the original observation mode is applied.

The learning unit 30 is a first learning unit that learns the CNN using a plurality of image groups having different observation modes, and a second learning unit that relearns the CNN using only the image groups to which the original observation mode is applied. Can be equipped.

<Action effect>
According to the image processing according to the fourth embodiment, the influence of the image group to which the observation mode different from the original is applied is suppressed after using the information of the image group to which the observation mode different from the original is applied. Learning can be carried out.

[Overall configuration of the endoscope system that acquires endoscopic images]
FIG. 8 is an overall configuration diagram of the endoscope system. The endoscope system 300 shown in FIG. 8 includes an endoscope 302, a light source device 311, a processor device 312, a display device 313, an image processing device 314, an input device 315, and a monitor device 316.

The endoscope 302 is an electronic endoscope. Further, the endoscope 302 is a flexible endoscope. The endoscope 302 includes an insertion unit 320, an operation unit 321 and a universal cord 322. The insertion unit 320 is inserted into the subject. The entire insertion portion 320 has a small diameter and is formed in a long shape.

The insertion portion 320 includes a soft portion 325, a curved portion 326, and a tip portion 327. The insertion portion 320 is configured by connecting a soft portion 325, a curved portion 326, and a tip portion 327 in succession. The soft portion 325 has flexibility in order from the proximal end side to the distal end side of the insertion portion 320. The curved portion 326 has a structure that can be bent when the operating portion 321 is operated. The tip portion 327 incorporates an image pickup optical system (not shown), an image pickup element 328, and the like.

A CMOS image sensor or a CCD image sensor is applied to the image sensor 328. CMOS is an abbreviation for Complementary Metal Oxide Semiconductor. CCD is an abbreviation for Charge Coupled Device.

An observation window (not shown) is arranged on the tip surface 327a of the tip portion 327. The observation window is an opening formed on the tip surface 327a of the tip portion 327. A cover (not shown) is attached to the observation window. An imaging optical system (not shown) is arranged behind the observation window. The image light of the observed portion is incident on the image pickup surface of the image pickup element 328 through the observation window, the image pickup optical system, and the like. The image sensor 328 captures the image light of the observed portion incident on the image pickup surface of the image sensor 328 and outputs an image pickup signal. The imaging referred to here includes the meaning of converting the reflected light from the observed portion into an electric signal.

The operation unit 321 is continuously provided on the base end side of the insertion unit 320. The operation unit 321 includes various operation members operated by the operator. Specifically, the operation unit 321 includes two types of curved operation knobs 329. The bending operation knob 329 is used when the bending operation of the bending portion 326 is performed. The operator may be referred to as a doctor, an operator, an observer, a user, or the like.

The operation unit 321 includes an air supply / water supply button 330 and a suction button 331. The air supply / water supply button 330 is used when the operator performs an air supply / water supply operation. The suction button 331 is used when the operator performs a suction operation.

The operation unit 321 includes a still image imaging instruction unit 332 and a treatment tool introduction port 333. The still image imaging instruction unit 332 is operated by the operator when capturing a still image of the observed portion. The treatment tool introduction port 333 is an opening for inserting the treatment tool into the inside of the treatment tool insertion passage through which the inside of the insertion portion 320 is inserted. It should be noted that the treatment tool insertion passage and the treatment tool are not shown.

The universal cord 322 is a connection cord for connecting the endoscope 302 to the light source device 311. The universal cord 322 includes a light guide 335 that inserts the inside of the insertion portion 320, a signal cable 336, and a fluid tube (not shown).

The tip of the universal cord 322 includes a connector 337a connected to the light source device 311 and a connector 337b branched from the connector 337a and connected to the processor device 312.

When the connector 337a is connected to the light source device 311, a light guide 335 and a fluid tube (not shown) are inserted into the light source device 311. As a result, necessary illumination light, water, and gas are supplied from the light source device 311 to the endoscope 302 via the light guide 335 and a fluid tube (not shown).

As a result, the illumination light is emitted from the illumination window (not shown) on the tip surface 327a of the tip portion 327 toward the observed portion. Further, in response to the pressing operation of the air supply / water supply button 330, gas or water is injected from the air supply / water supply nozzle (not shown) on the tip surface 327a of the tip portion 327 toward the observation window (not shown) on the tip surface 327a.

When the connector 337b is connected to the processor device 312, the signal cable 336 and the processor device 312 are electrically connected. As a result, the image pickup element 328 of the endoscope 302 outputs the image pickup signal of the observed portion to the processor device 312, and the processor device 312 outputs the control signal to the endoscope 302 via the signal cable 336.

In the present embodiment, a flexible endoscope has been described as an example of the endoscope 302, but as the endoscope 302, various electrons capable of capturing a moving image of an observed part such as a rigid endoscope are used. An endoscope may be used.

The light source device 311 supplies illumination light to the light guide 335 of the endoscope 302 via the connector 337a. As the illumination light, white light or light in a specific wavelength band can be applied. The illumination light may be a combination of white light and light in a specific wavelength band. The light source device 311 is configured so that light in a wavelength band according to the purpose of observation can be appropriately selected as illumination light.

The white light may be either light in a white wavelength band or light in a plurality of wavelength bands. The specific wavelength band is narrower than the white wavelength band. As the light of a specific wavelength band, light of one kind of wavelength band may be applied, or light of a plurality of wavelength bands may be applied. A particular wavelength band is sometimes referred to as special light.

The processor device 312 controls the operation of the endoscope 302 via the connector 337b and the signal cable 336. Further, the processor device 312 acquires an image pickup signal from the image pickup element 328 of the endoscope 302 via the connector 337b and the signal cable 336. The processor device 312 applies a predetermined frame rate to acquire an image pickup signal output from the endoscope 302.

The processor device 312 generates an endoscope image, which is an observation image of the observed portion, based on the image pickup signal acquired from the endoscope 302. The endoscopic image 338 referred to here includes a moving image. The endoscopic image 338 may include a still image 339.

When the still image imaging instruction unit 332 of the operation unit 321 is operated, the processor device 312 generates a still image 339 of the observed portion based on the imaging signal acquired from the imaging element 328 in parallel with the generation of the moving image. .. The still image 339 may be generated at a higher resolution than the resolution of the moving image.

When generating the endoscopic image 338, the processor device 312 corrects the image quality by applying digital signal processing such as white balance adjustment and shading correction. The processor device 312 may add incidental information specified in the DICOM standard to the endoscopic image 338. DICOM is an abbreviation for Digital Imaging and Communications in Medicine.

The endoscopic image 338 is an in-vivo image obtained by imaging the inside of a subject, that is, the inside of a living body. When the endoscopic image 338 is an image obtained by imaging using light in a specific wavelength band, both are special optical images. Then, the processor device 312 outputs the generated endoscopic image 338 to the display device 313 and the image processing device 314, respectively. The processor device 312 may output the endoscopic image 338 to a storage device (not shown) via a network (not shown) according to a communication protocol conforming to the DICOM standard.

The display device 313 is connected to the processor device 312. The display device 313 displays the endoscopic image 338 transmitted from the processor device 312. The operator can move the insertion unit 320 forward and backward while checking the endoscopic image 338 displayed on the display device 313. When a lesion or the like is detected at the observed site, the operator can operate the still image imaging instruction unit 332 to capture a still image of the observed site.

A computer is used as the image processing device 314. As the input device 315, a keyboard, a mouse, or the like that can be connected to a computer is used. The connection between the input device 315 and the computer may be either a wired connection or a wireless connection. As the monitor device 316, various monitors that can be connected to a computer are used.

As the image processing device 314, a diagnostic support device such as a workstation and a server device may be used. In this case, the input device 315 and the monitor device 316 are provided for each of a plurality of terminals connected to a workstation or the like. Further, as the image processing device 314, a medical service support device that supports the creation of a medical report or the like may be used.

The image processing device 314 acquires the endoscopic image 338 and stores the endoscopic image 338. The image processing device 314 controls the reproduction of the monitor device 316. The image processing device 314 shown in FIG. 8 may include the function of the endoscopic image learning device 10 described with reference to the drawings 1 to 7. Further, the input device 315 shown in FIG. 8 corresponds to the operation unit 18 shown in FIG. The monitor device 316 shown in FIG. 8 corresponds to the display unit 26 shown in FIG.

The term "image" in the present specification includes the meaning of a still image 339 such as an electric signal representing an image and information representing an image. The term image in the present specification means at least one of the image itself and the image data.

Also, the term image storage can be read as image storage. Image memory here means non-temporary image storage. The image processing device 314 may include a memory for temporary storage for temporarily storing an image.

The input device 315 is used for inputting an operation instruction to the image processing device 314. The monitor device 316 displays the endoscopic image 338 under the control of the image processing device 314. The monitor device 316 may function as a display unit for various information in the image processing device 314.

The image processing device 314 may be connected to a storage device (not shown) via a network (not shown). For the image storage format and communication between each device via the network, a protocol compliant with the DICOM standard and the DICOM standard can be applied.

A storage device (not shown) can be applied with a storage or the like that stores data non-temporarily. The storage device may be managed by using a server device (not shown). The server device can be applied to a computer that stores and manages various data.

[Observation mode of endoscopic system]
FIG. 9 is a functional block diagram of the endoscopic system. The endoscope system 300 is configured to be able to switch between a normal light observation mode and a special light observation mode. The operator can operate an observation mode switching button (not shown) to switch between the normal light observation mode and the special light observation mode.

The light source device 311 includes a first laser light source 400, a second laser light source 402, and a light source control unit 404. The first laser light source 400 is a blue laser light source having a center wavelength of 445 nm. The second laser light source 402 is a purple laser light source having a center wavelength of 405 nm. A laser diode can be used as the first laser light source 400 and the second laser light source 402. The light emission of the first laser light source 400 and the second laser light source 402 is individually controlled by using the light source control unit 404. The emission intensity ratio between the first laser light source 400 and the second laser light source 402 can be freely changed.

The endoscope 302 includes a first optical fiber 410, a second optical fiber 412, a phosphor 414, a diffusion member 416, an image pickup lens 418, an image pickup element 328, and an analog-to-digital converter 420.

The irradiation unit is configured by using the first laser light source 400, the second laser light source 402, the first optical fiber 410, the second optical fiber 412, the phosphor 414, and the diffusion member 416.

The laser light emitted from the first laser light source 400 is applied to the phosphor 414 arranged at the tip end portion 327 of the endoscope 302 via the first optical fiber 410. The phosphor 414 is composed of a plurality of types of phosphors that absorb a part of the blue laser light from the first laser light source 400 and excite and emit light in a range from green to yellow. As a result, the light emitted from the phosphor 414 is the blue one that is transmitted without being absorbed by _{the excitation light L 11 in the} range from green to yellow, which uses the blue laser light from the first laser light source 400 as the excitation light, and the phosphor 414. The laser light L ₁₂ is combined to form a white or pseudo-white light L ₁ .

It should be noted that the white light referred to here is not limited to that strictly includes all wavelength components of visible light. For example, light containing a specific wavelength band such as R, G, and B may be used, and light containing a wavelength component from green to red or light containing a wavelength component from blue to green is also broadly defined. It shall include.

On the other hand, the laser light emitted from the second laser light source 402 is applied to the diffusion member 416 arranged at the tip end portion 327 of the endoscope 302 via the second optical fiber 412. As the diffusion member 416, a translucent resin material or the like can be used. The light emitted from the diffusing member 416 becomes _{light L 2} having a narrow band wavelength in which the amount of light is uniform in the irradiation region.

FIG. 10 is a graph showing the light intensity distribution. The light source control unit 404 changes the light amount ratio between the first laser light source 400 and the second laser light source 402. Thus, the light quantity ratio between the light L ₁ and the light L ₂ is changed, the changed wavelength pattern of the light L ₁ and the irradiation light L ₀ is a composite light of the light L ₂ is, wavelengths different patterns depending on the observation mode The irradiation light L ₀ can be irradiated.

The endoscope system 300 shown in FIG. 9 includes an image pickup unit including an image pickup lens 418, an image pickup element 328, and an analog-to-digital conversion unit 420. As shown in FIG. 8, the imaging unit is arranged at the tip portion 327 of the endoscope 302.

The image pickup lens 418 shown in FIG. 9 forms an image of the incident light on the image pickup element 328. The image sensor 328 generates an analog signal according to the received light. The analog signal output from the image sensor 328 is converted into a digital signal by the analog-digital conversion unit 420 and input to the processor device 312.

The processor device 312 includes an image pickup control unit 440, an image processing unit 442, an image acquisition unit 444, and an image recognition unit 446.

The image pickup control unit 440 controls the light source control unit 404 of the light source device 311, the image pickup element 328 and the analog-digital conversion unit 420 of the endoscope 302, and the image processing unit 442 of the processor device 312, and uses the endoscope system 300. It controls the imaging of moving images and still images.

The image processing unit 442 performs image processing on the digital signal input from the analog-to-digital conversion unit 420 of the endoscope 302 to generate an image. The image processing unit 442 performs image processing according to the wavelength pattern of the irradiation light at the time of imaging.

The image acquisition unit 444 acquires the image generated by the image processing unit 442. That is, the image acquisition unit 444 sequentially acquires a plurality of images captured in time series by applying a constant frame rate to the inside of the body cavity of the subject. The image acquisition unit 444 may acquire an image input from the input unit 447 or an image stored in the storage unit 468. Further, the image may be acquired from an external device such as a server connected to a network (not shown). The images in these cases are also preferably a plurality of images captured in time series.

The image recognition unit 446 uses the learning model learned by the endoscopic image learning device 10 to perform image recognition of the image acquired by the image acquisition unit 444. In the present embodiment, the lesion is recognized from the image acquired by the image acquisition unit 444. Lesions are not limited to those caused by disease and include areas of a state that is different from the apparently normal state.

Examples of lesions include polyps, cancer, diverticula of the large intestine, inflammation, treatment scars, clip sites, bleeding points, perforations, and vascular atypia. Examples of treatment scars include EMR scars and ESD scars. EMR is an abbreviation for Endoscopic Mucosal Resection. ESD is an abbreviation for Endoscopic Submucosal Dissection.

The display control unit 450 causes the display device 313 to display the image generated by using the image processing unit 442. The display control unit 450 may superimpose and display the lesion recognized by the image recognition unit 446 on the image so as to be recognizable.

The storage control unit 452 stores the image generated by using the image processing unit 442 in the storage unit 468. For example, and stores the information of the wavelength pattern of the irradiation light L ₀ at the time of capturing a captured image and the image according to the acquired instruction of a still image in the storage unit 468.

An example of the storage unit 468 is a storage device such as a hard disk. The storage unit 468 is not limited to the one built in the processor device 312. For example, it may be an external storage device (not shown) connected to the processor device 312. The external storage device may be connected via a network (not shown).

The endoscope system 300 configured in this way normally captures a moving image at a constant frame rate, and displays the captured image on the display device 313. In addition, a lesion is detected from the captured moving image, and the detected lesion is recognizable and superimposed on the moving image and displayed on the display device 313.

According to the endoscope system 300, an image recognition unit 446 using a learning model learned by the endoscope image learning device 10 is applied to automatically recognize an endoscope image, and perform image recognition of a special optical image. Can be done properly.

The endoscope system 300 described with reference to FIGS. 8 to 10 can acquire a normal light image 14a and a special light image 16a.

[Modification example of endoscopic system]
[Modification example of processor device]
The processor device 312 may have the function of the image processing device 314. That is, the processor device 312 may be integrally configured with the image processing device 314. In such an embodiment, the display device 313 can also be used as the monitor device 316. The processor device 312 may include a connection terminal to which the input device 315 is connected.

[Example of generating a feature image]
The medical image is based on at least one of a white band light, a normal image obtained by irradiating light in a plurality of wavelength bands as light in the white band, and a special light image obtained by irradiating light in a specific wavelength band. An operation can be used to generate a feature image.

[Example of application to a program that makes a computer function as an image processing device]
The above-mentioned endoscopic image learning method can be configured as a program that realizes a function corresponding to each step in the endoscopic image learning method by using a computer. For example, a computer is provided with an image acquisition function for acquiring a first image and a second image having different spectral distributions, an image processing function for performing image processing on the first image to generate a third image, and a plurality of first images. Each of the first image group including, the second image group including a plurality of second images, the third image group including a plurality of third images, and the first image group, the second image group, and the third image group, respectively. It is a program that realizes a learning function that learns a recognizer to be applied to automatic recognition using correct answer data that is the correct recognition result, and the image processing function is the second of the bands included in the spectral distribution of the first image. Image processing that suppresses signals in a band whose characteristics are different from the corresponding band of the image, and signals in a band that has the same characteristics as the corresponding band in the second image or a signal in a similar band among the bands included in the first image. A program that generates a third image from a first image can be realized by performing at least one of the image processing to be emphasized.

To store the program for realizing the above-mentioned endoscopic image learning function in a computer in a computer-readable information storage medium, which is a tangible non-temporary information storage medium, and provide the program through the information storage medium. Is possible.

Further, instead of the mode in which the program is stored and provided in the non-temporary information storage medium, the mode in which the program signal is provided via the network is also possible.

[Combination of Embodiments and Modifications]
The components described in the above-described embodiment and the components described in the modified examples can be used in combination as appropriate, or some components can be replaced.

In the embodiment of the present invention described above, the constituent requirements can be appropriately changed, added, or deleted without departing from the spirit of the present invention. The present invention is not limited to the embodiments described above, and many modifications can be made by a person having ordinary knowledge in the art within the technical idea of the present invention.

10 Endoscopic image learning device 12 Communication unit 14 First storage device 14a Normal light image 16 Second storage device 16a Special optical image 17 Third storage device 17a Processing image 18 Operation unit 20 CPU
22 RAM
24 ROM
26 Display unit 30 Learning unit 32 CNN
32a Input layer 32b Intermediate layer 32c Output layer 32d Folding layer 32e Pooling layer 32f Set 32g Fully coupled layer 34 Error calculation unit 36 Parameter update unit 40 Image generation unit 100 Spectral sensitivity distribution 102 B channel component 104 G channel component 106 R channel component 120 Intensity distribution of normal light 140 Spectral distribution characteristic of normal light image 142 B channel component 144 G channel component 146 R channel component 160 Intensity distribution of illumination light in LCI 180 Spectral distribution characteristic of special light image 182 B channel component 184 G channel component 186 R channel component 200 Spectral distribution characteristic of normal light image 202 B channel component 204 G channel component 206 R channel component 220 Spectral distribution characteristic of special light image 222 B channel component 224 G channel component 300 Endoscope system 302 Endoscope 311 Light source Device 312 Processor device 313 Display device 314 Image processing device 315 Input device 316 Monitor device 320 Insertion unit 321 Operation unit 322 Universal cord 325 Flexible part 326 Curved part 327 Tip part 327a Tip surface 328 Image sensor 329 Curved operation knob 330 Air supply / water supply button 331 Suction button 332 Still image imaging instruction unit 333 Treatment tool introduction port 335 Light guide 336 Signal cable 337a Connector 337b Connector 338 Endoscope image 339 Still image 400 First laser light source 402 Second laser light source 404 Light source control unit 410 First light Fiber 412 Second optical fiber 414 Fluorescent material 416 Diffusing member 418 Imaging lens 420 Analog digital conversion unit 442 Image processing unit 444 Image acquisition unit 446 Image recognition unit 447 Input unit 450 Display control unit 452 Storage control unit 468 Storage unit S10 to S21 Each step of the spectroscopic image learning method

Claims (18)

An image acquisition unit that acquires the first image and the second image whose spectral distributions are different from each other,
An image processing unit that performs image processing on the first image to generate a third image,
A first image group including the plurality of the first images, a second image group including the plurality of the second images, a third image group including the plurality of the third images, and the first image group and the second image. A learning unit that learns a recognizer to be applied to automatic recognition using the correct answer data of each of the image group and the third image group.
With
The image processing unit suppresses signals in a band having characteristics different from the corresponding band of the second image among the bands included in the spectral distribution of the first image, and the band included in the first image. A medical image that generates the third image from the first image by performing at least one of image processing that emphasizes a signal in a band having the same characteristics as the corresponding band of the second image or a signal in a similar band. Learning device. The medical image learning device according to claim 1, wherein the image processing unit performs a process of converting the luminance dispersion on the first image. The medical image learning device according to claim 1, wherein the image processing unit performs a process of changing the brightness of the first image to a predetermined brightness value. The medical image learning device according to claim 1, wherein the image processing unit performs a process of changing the brightness of the first image to a random brightness value. The first image is a normal light image captured by using normal light.
The medical image learning device according to claim 1, wherein the second image is a special light image captured by using special light having a narrower band than normal light. The medical image learning device according to claim 5, wherein the image processing unit performs a process of suppressing a B channel component on the normal optical image. The medical image learning apparatus according to claim 5, wherein the image processing unit performs a process of emphasizing at least one of an R channel component and a G channel component on the normal optical image. The medical image learning device according to claim 5, wherein the image processing unit performs a process of suppressing an R channel component on the normal optical image. The medical image learning apparatus according to claim 5, wherein the image processing unit performs a process of emphasizing a B channel component and a G channel component on the normal optical image. The image processing unit performs a process of suppressing a specified frequency component of the spatial frequency in the processing target band of the first image, or a process of emphasizing the specified frequency component of the spatial frequency in the processing target band of the first image. The medical image learning device according to claim 1. The first image is a normal light image captured by using normal light.
The second image is a special light image taken by using special light having a narrower band than normal light.
The medical image learning apparatus according to claim 10, wherein the image processing unit performs a process of emphasizing a high frequency component of a B channel component of the normal optical image. The medical image learning device according to any one of claims 1 to 11, wherein the learning unit switches a learning method between learning of the first image group and learning of the second image group. The medical image learning device according to any one of claims 1 to 11, wherein the learning unit switches a learning method between learning of the second image group and learning of the third image group. The medical image learning device according to claim 12 or 13, wherein the learning unit applies a learning method that suppresses or emphasizes the influence of the third image on learning. The learning unit is a recognizer that has been trained using the first image group, the third image group, the correct answer data of the first image group, and the correct answer data of the third image group, or the third image group. The invention according to claim 12 or 13, wherein the trained recognizer using the correct answer data of the third image group is trained using the correct answer data of the second image group and the second image group. Medical image learning device. An image acquisition process for acquiring a first image and a second image having different spectral distributions from each other,
An image processing step of performing image processing on the first image to generate a third image, and
A first image group including the plurality of the first images, a second image group including the plurality of the second images, a third image group including the plurality of the third images, and the first image group and the second image. A learning process for learning a recognizer to be applied to automatic recognition using the correct answer data of each of the image group and the third image group.
Including
The image processing step includes image processing that suppresses signals in a band having characteristics different from the corresponding band of the second image among the bands included in the spectral distribution of the first image, and the band included in the first image. A medical image that generates the third image from the first image by performing at least one of image processing that emphasizes a signal in a band having the same characteristics as the corresponding band of the second image or a signal in a similar band. Learning method. On the computer
Image acquisition function to acquire the first image and the second image whose spectral distributions are different from each other,
An image processing function that performs image processing on the first image to generate a third image, a first image group including the plurality of the first images, a second image group including the plurality of the second images, and a plurality of. Learning to learn a recognizer to be applied to automatic recognition by using the correct answer data of each of the third image group including the third image, the first image group, the second image group, and the third image group. It is a program that realizes the function
The image processing function includes image processing that suppresses signals in a band having characteristics different from the corresponding band of the second image among the bands included in the spectral distribution of the first image, and a band included in the first image. A program that generates the third image from the first image by performing at least one of image processing that emphasizes a signal in a band having the same characteristics as the corresponding band of the second image or a signal in a similar band. A non-temporary, computer-readable storage medium when the instructions stored in the storage medium are read by a computer.
An image acquisition function that acquires the first and second images with different spectral distributions from each other,
An image processing function that performs image processing on the first image to generate a third image,
A first image group including the plurality of the first images, a second image group including the plurality of the second images, a third image group including the plurality of the third images, and the first image group and the second image. It includes a learning function for learning a recognizer to be applied to automatic recognition using the correct answer data of each of the image group and the third image group.
The image processing function includes image processing that suppresses signals in a band having characteristics different from the corresponding band of the second image among the bands included in the spectral distribution of the first image, and a band included in the first image. An image in which the third image is generated from the first image by performing at least one of image processing for emphasizing a signal in a band having the same characteristics as the corresponding band of the second image or a signal in a similar band. A storage medium that causes a computer to perform learning functions.

Images