JP6528608B2 - Diagnostic device, learning processing method in diagnostic device, and program - Google Patents

Description

  The present invention relates to a diagnostic device, a learning processing method in the diagnostic device, and a program.

  As a diagnosis of skin lesions, visual inspection is always performed, and a lot of information can be obtained. However, with the naked eye or the loupe alone, it is difficult even to distinguish between hokuro and stains, and it is also difficult to distinguish between benign tumors and malignant tumors. Therefore, dermoscopy diagnosis is performed in which a lesion is photographed using a dermoscope camera, but the present situation is that the identification of a case by observation of an image depends on the skill of a doctor.

  The identification of the image by the conventional dermoscopy image is processed based on the original image, so that the doctor can not focus on the blood vessel structure of the affected area which is a clue for identifying the case, and thus the case pattern There was a problem with the accuracy of identification.

  As a general machine learning method for pattern identification, there is known an ensemble learning method in which discrimination is performed with a plurality of classifiers and the discrimination results are integrated to perform discrimination with high accuracy. The ensemble learning method is based on the principle of discrimination that the variance of estimated values becomes small when a classifier with a large estimated variance value is gathered by a plurality of classifiers to perform discrimination by majority.

  For example, Patent Document 1 discloses a method of performing ensemble learning by preferentially selecting a basis function effective for pattern identification from among the innumerable choices of basis functions in the feature amount space.

JP 2012-43156 A

  According to the technology disclosed in Patent Document 1 described above, although the amount of calculation and the amount of data at the time of learning can be suppressed as compared with the conventional ensemble method, problems still remain in the accuracy of identification.

  The present invention has been made to solve the above-described problems, and a diagnostic device capable of performing diagnosis support with higher accuracy by improving the accuracy of discrimination by an ensemble classifier, and a learning processing method in the diagnostic device, and The purpose is to provide a program.

In order to solve the above-described problems, one aspect of the present invention is a diagnostic device for diagnosing a skin disease using a skin image, which is based on a plurality of unknown skin image data related to a subject to be diagnosed. And an ensemble discriminator for identifying whether the subject is a disease or not, wherein the ensemble discriminator comprises: original image data relating to the subject; and the first image data subjected to a predetermined first conversion process on the original image data. converted image data, and, of the second converted image data by the first conversion processing different from the second conversion processing on the original image data has been subjected, at least the second converted image data In order to correspond to the plurality of unknown skin image data including two included , at least two unit discriminators and a plurality of discrimination values separately obtained by the unit discriminators are integrated to obtain a final judgment value. Judgment Comprising a stage, and in that said second conversion processing, on the basis of the luminance component and color information components of the original image data, a process of extracting highlights the portion corresponding to the blood vessel from the original image data It features.
Other features of the present invention will become apparent from the description of the present specification and the accompanying drawings.

  According to the present invention, it is possible to provide a diagnostic device, a learning processing method in a diagnostic device, and a program capable of improving the accuracy of identification by an ensemble classifier and performing diagnosis support with higher accuracy.

It is a block diagram showing composition of a diagnostic device concerning a 1st embodiment of the present invention. It is an operation conceptual diagram of the processing unit of FIG. It is a flowchart which shows the basic operation | movement of the diagnostic apparatus of FIG. It is a flowchart which shows the flow of the machine learning discriminator production | generation process of FIG. It is a flowchart which shows an example (image conversion process A) of the process which converts an image into the learning skin image of FIG. It is a figure which shows an example of the process which obtains a blood-vessel candidate image from the luminance image of FIG. It is a figure which shows an example of the process which extracts the blood vessel likeness of FIG. 6 as likelihood A. FIG. It is a figure which shows the other example of the process which extracts the blood vessel likeness of FIG. 6 as likelihood A. FIG. It is a flowchart which shows the other example (image conversion process B) of the process which converts an image into the learning skin image of FIG. It is a figure which shows an example of the process which performs blood vessel extraction from likelihood A which shows the blood vessel likeness of FIG. It is a flowchart which shows the flow of a test image (unknown image identification) identification process of FIG. It is a block diagram showing composition of a diagnostic device concerning a 2nd embodiment of the present invention. It is the figure which showed the identification evaluation result in tabular form.

  Hereinafter, with reference to the accompanying drawings, modes (hereinafter, embodiments) for carrying out the present invention will be described in detail. In the following drawings, the same elements are denoted by the same reference numerals or symbols throughout the description of the embodiments.

(Configuration of the first embodiment)
FIG. 1 is a block diagram showing the configuration of a diagnosis apparatus 100A according to the first embodiment of the present invention. As shown in FIG. 1, a photographing apparatus 110 with a dermoscope is connected to the diagnostic apparatus 100A according to the first embodiment of the present invention.

  The dermoscope-equipped imaging device 110 captures an image according to an instruction from the diagnostic device 100A, stores the captured image (dermoscopy image) in the image storage unit 104, and displays the image on the display device 120. The photographed image is subjected to image processing by the diagnostic device body 10 (the pre-processing unit 10 a and the processing unit 11 a), stored in the image storage unit 104, and displayed on the display device 120.

  The input device 130 performs a photographing start instruction of a dermoscope image, an operation for selecting a region in a dermoscopy image described later, and the like. Note that the display device 120 is configured of, for example, an LCD (Liquid Crystal Display) monitor or the like, and the input device 130 is configured of a mouse or the like.

  The learning skin image storage unit 103 is a skin image database in which the identification name of a disease given for learning and known skin image data are associated with each other and recorded.

  The diagnostic device body 10 includes a preprocessing unit 10a, a processing unit 11a (an ensemble discriminator), and an acquisition unit 12a.

  The pre-processing unit 10 a converts the captured image (original image data) stored in the image storage unit 104 by performing the structure clear conversion process, or the converted image data by performing the site enhancement process on the original image data. Are generated and output to the processing unit 11a. Here, the clear structure conversion process is a process of emphasizing the luminance component of the original image data, and the part emphasizing process is a process of extracting a corresponding part from the luminance component or the luminance component and the color information component of the original image data. It is. The converted image data may be rotated or inverted.

  For this reason, the preprocessing unit 10 a includes a separation unit 101 and an extraction unit 102.

  The separation unit 101 functions as a unit that separates a photographed image into a luminance component and a color information component.

  The extraction means 102 functions as a means for extracting a region to be diagnosed, and is a color space constituted by a first extraction means 102a for extracting a region candidate by a luminance component, and a region likeness from a luminance component and a color information component. And at least one of the second extraction means 102b for performing extraction, and performs morphological processing including smoothing filter processing on the extracted region candidate or region likeness.

  When the extraction means 102 extracts a shape indicating a part candidate or part likeness in a structured element in a photographed image, the first extraction means 102a extracts a part candidate by a first morphological process using a luminance component. The second extracting unit 102b may extract the region likeness using the color space, and the extracting unit 102 may integrate the extracted region candidate and the region likeness to generate an extracted image.

  When the extraction unit 102 extracts a part candidate in the captured image or a shape indicating a part likeness in the structured element, the second extraction unit 102 b extracts the likeness using the color space, and the extraction unit 102 is extracted. The region extracted image may be generated by the second morphological process using the region likeness.

  Here, in the first morphological processing, the extracted luminance component is subjected to closing processing in which expansion processing and contraction processing are repeated in this order, smoothing filter processing on the luminance component subjected to the closing processing, and smoothing filter processing. And subtracting the luminance component of the photographed image from the luminance component. In the second morphological process, an opening process in which contraction and expansion processes are repeated in this order on the extracted part likeness, a smoothing filter process on the part likeness subjected to the opening process, and a smoothing filter process are performed It includes a process of subtracting the site likeness from the extracted site likeness.

  The processing unit 11a includes an ensemble discriminator that determines whether or not there is a disease based on a plurality of unknown skin image data related to an object to be diagnosed that has been preprocessed by the preprocessing unit 10a. Corresponds to a plurality of skin image data including at least two of original image data relating to the object, (first converted image data) converted from the original image data, and the same (second converted image data) And at least two unit discriminators 111 (CNN1), 112 (CNN2)... And a judging means 113 for integrating final identification values obtained by the unit discriminators 111 and 112 to obtain final judgment values. Including.

  Note that the unit classifiers 111, 112 have a convolutional neural network (CNN: Convolution Neural Network) that learns based on a plurality of known skin image data related to a disease, and the conversion generated by the preprocessing unit 10a. Image data is input to this convolutional neural network in advance to be learned, and functions as a classifier that generates classification information so that the disease to be diagnosed can be identified.

  Note that, for example, the unit identifiers 111, 112,... May be given learning in advance in the manufacturing stage before the diagnostic device 100A is shipped from the manufacturing factory, or after being shipped in advance by the hospital etc. It may be attached to learning. Here, "in advance" means before identifying the disease to be diagnosed.

  FIG. 2 shows a representative configuration of a convolutional neural network (CNN). According to FIG. 2, in the convolutional neural network, a plurality of known skin image data (converted image data) are input in the learning stage, and a plurality of unknown skin image data (converted image data) are input in the testing stage. And an intermediate layer 111 b having a plurality of sets composed of a convolution layer and a pooling layer and extracting features from a plurality of known skin image data or a plurality of unknown skin image data, and a diagnosis target based on the extracted features And an output layer 111c that outputs an identification value for each of the categories.

  The processing of the above-described convolutional neural network is performed via a plurality of processing units a connected in multiple stages. The input and output of each processing unit a are a plurality of two-dimensional images represented by a feature map b which is a plurality of features extracted from an input image. In this case, the input image is also regarded as one feature amount map. Here, a pair of units of convolution operation and pooling are connected in multiple stages as a processing unit a to calculate a feature quantity vector. An identification process is performed on the feature amount vector by the determination unit 113 described later to obtain an output class.

  The determination means 113 inputs the extracted feature and performs identification. In the learning of the convolutional neural network, the weight of each layer is updated by learning by the error reverse propagation method. A multilayer perceptron is used as an identification process. The multilayer perceptron is a non-linear class discriminator composed of an input layer 111a, an intermediate layer 111b, and an output layer 111c. The weight between each layer is obtained by the stochastic gradient descent method by the error propagation method. At the time of identification, feature quantities are sequentially propagated, and the output of each unit in the output layer is classified as a posterior probability of each class. Here, the identification values obtained separately for each unit identifier by 111 and 112 are integrated, for example, by averaging to obtain the final judgment value.

  A convolutional neural network is a general method for classifying images with high accuracy, and is described in detail, for example, on the Internet URL (http://en.wikipedia.org/wiki/Convolutional neural network). . A convolutional neural network (CNN) is a type of deep learning (deep neural network; deep neural network) in which neural networks that simulate neural networks of the brain are learned in multiple layers and can be suitably used for image analysis. In addition, it is also possible to adopt other methods of deep learning or to combine with other methods.

  The acquiring unit 12a is capable of acquiring a plurality of unknown skin image data, and outputs the plurality of unknown skin image data to the preprocessing unit 10a for image conversion such as clear structure and region emphasis.

(Operation of the first embodiment)
Hereinafter, the operation of the diagnostic device 100A according to the first embodiment of the present invention shown in FIGS. 1 and 2 will be described in detail with reference to flowcharts in FIG. The following operations can be configured as a learning processing program to be executed by a computer.

  In FIG. 3, the acquiring unit 12a of the diagnostic device body 10 first acquires a plurality of unknown skin image data related to a diagnosis target as a learning skin image (step S10: learning image collection). Specifically, when collecting unknown skin image data, the doctor performs dermoscopy photographing of the diseased part of the patient, and the acquiring unit 12a takes in a plurality of unknown skin image data by the photographing operation, and the preprocessing unit 10a Output to

  At this time, the preprocessing unit 10a registers the disease name of the case in the learning skin image storage unit 103 in association with the collected learning skin image data. Then, the pre-processing unit 10a determines whether the necessary number of learning skin images have been collected, repeatedly executes the above procedure until the required number of sheets are collected, and the skin image database is stored on the learning skin image storage unit 103. To construct.

  After the acquisition unit 12a executes unknown skin image data collection processing in step S10, the preprocessing unit 10a executes image conversion processing such as clearness of structure, region emphasis, etc. on unknown skin image data, and further 90 degrees. Data increase processing such as rotation is performed and output to the processing unit 11a. The processing unit 11a extracts the feature of the learning skin image which is the input image by the unit classifiers 111, 112,... Repeating the convolution operation and the pooling by the weight filter (step S20: machine learning device generation) processing).

  Specifically, the unit classifiers 111, 112,... Raster scan the weight image with respect to the input image and repeatedly perform the convolution operation to obtain the feature map b (FIG. 2). Then, pooling is performed, and a discriminator generation process is executed to output values from a small area of the m-1 feature map b and convert the values into m feature maps.

  Next, the determination means 113 performs identification by inputting the features extracted by the unit classifiers 111, 112,. A multilayer perceptron is used as a discrimination process, and at the time of discrimination, the feature amount is sequentially propagated, and the discrimination process is performed on an unknown image which classifies an input image with the output of each unit of the output layer 111c as a posterior probability of each class Step S30: test image identification).

  FIG. 4 is a flowchart showing the procedure of the machine learning discriminator generation process (step S20 of FIG. 3) of the diagnosis apparatus 100A according to the first embodiment of the present invention. According to FIG. 4, first, the preprocessing unit 10a executes an image conversion process for site enhancement and structure clarity on the learning skin image (step S201). Here, region emphasis is image data obtained by performing region emphasis processing such as blood vessels on original image data. The part emphasizing transformation process is a process of extracting a corresponding part from the luminance component or the luminance component and the color information component of the original image data, and the structure clarity is a process of emphasizing the luminance component of the image data.

  Hereinafter, with reference to FIGS. 5-10, the detailed process sequence of "conversion to the skin image for learning" of FIG.4 S201 is demonstrated.

  In the image conversion, the preprocessing unit 10a first acquires a photographed image of an affected area, for example, a lesion site of the skin, photographed by the photographing apparatus 110 with a dermoscope. Then, the acquired captured image is stored in a predetermined area of the image storage unit 104. Subsequently, the preprocessing unit 10a performs a blood vessel extraction process from the captured image and executes a process of emphasizing the extracted blood vessel.

  A detailed processing procedure for image conversion (image conversion processing A) is shown in FIG. According to FIG. 5, the preprocessing unit 10a first converts the captured image in the RGB color space into the Lab color space (more precisely, the CIE 1976 L * a * b color space) by the separation unit 101 (step S201a). The details of Lab color space, Internet URL (http://en.wikipedia.org/wiki/Lab%E8%89%B2%E7%A9%BA%E9%96%93) <September 1, 2014 It is described in>.

  Next, the preprocessing unit 10 a extracts the part selected as the target of diagnosis by the extraction unit 102. Specifically, the first extraction unit 102a extracts the candidate (blood vessel candidate) of the selected site from the luminance components separated in the Lab color space. Therefore, the first extraction unit 102a obtains the blood vessel candidate image BH by the morphological processing A (first morphological processing) using the L image corresponding to the luminance in the Lab color space subjected to color space conversion by the separation unit 101. (Step S202a). Here, the morphological processing is to apply a structuring element to an input image to generate a blood vessel candidate image BH as an output image of the same size, and each value of the output image is a corresponding pixel in the input image. Based on the comparison of the pixel with the neighboring pixels.

  The most basic morphological processes are expansion and contraction. Dilation adds pixels to object boundaries in the input image, and contraction removes boundary pixels. The number of pixels added to or deleted from an object depends on the size and shape of the structuring element used for image processing.

  Here, a method of executing morphology processing A and extracting a site (blood vessel candidate) selected as a target of diagnosis from the luminance component will be described. The bottom hat processing is shown in FIG. 6 in detail.

  According to FIG. 6, the first extraction unit 102a performs expansion processing on the L image to obtain a luminance image L1 after processing (step S202a-1). The details of expansion processing, Internet URL (http://www.mathworks.co.jp/jp/help/images/morphology-fundamentals-dilation-and-erosion.html) <September 1, 2014 reading> Have been described.

  Next, the first extraction unit 102a performs contraction processing on the luminance image L1 after expansion processing, and obtains a luminance image L2 after contraction processing (step S202a-2). Subsequently, the first extraction unit 102a performs smoothing filter processing to smooth the luminance on the luminance image L2 after the contraction processing, and obtains a smoothed luminance image L3 (step S202a-3). Here, smoothing is performed by a Gaussian filter.

The smoothing by the Gaussian filter is expressed by the following equation.
f (x, y) = (1 / √ (2πσ ^ 2)) exp (-(x ^ 2 + y ^ 2) / (2σ ^ 2))

In the Gaussian filter, weighting by Gaussian distribution is used as the predetermined rate. The degree of smoothness can be controlled by the magnitude of σ in the above-described arithmetic expression, and this is realized by setting a predetermined value. The smoothing filter is not limited to the Gaussian filter, and a median filter, an average filter or the like may be used. From the luminance image L3 after the smoothing process by subtracting the L image (BH = L 3 -L) to obtain an image BH after bottom-hat processing (Stiff S202a-4).

  The above process is repeated for the specified number of times, and when the specified number of times is completed, the image BH is taken as the blood vessel extraction image E. When the number is within the specified number, the expansion process (step S202a-1), the contraction process (step S202a-2), and the like are repeated using the image BH as the image L.

  Here, the expansion process is supplemented. For example, consider a structuring element of radius 5 dots. The expansion processing refers to performing processing of setting the maximum value within the range of the structuring element of the pixel of interest as the value of the image of interest for all pixels. That is, the target pixel value to be output is the maximum value of all the pixels in the vicinity of the input pixel. On the other hand, in the reduction process, the minimum value within the range of the structuring element of the pixel of interest is taken as the value of the pixel of interest. That is, the target pixel value is the minimum value of all pixels in the vicinity of the input pixel. Although the structuring element is a circle, it may be, for example, a rectangle. However, using a circle can reduce the smoothness of the smoothing filter.

  The description is returned to the flowchart of FIG. Next, the processing unit 11a causes the second extraction unit 102b to extract the depth (blood vessel likeness) of the selected portion from the color space configured from the luminance component and the color information component. Therefore, the second extraction unit 102b calculates the likelihood of a blood vessel as the likelihood A (step S203a). An example of how to determine the likelihood A is shown in FIG.

  According to the flowchart of FIG. 7, the preprocessing unit 10a causes the second extraction unit 102b to generate the a-axis value, which is the color information component according to the red hue direction of the Lab color space, and the bluish hue direction. It extracts using the value of the b-axis which is the color information component according to (step S203a-1). That is, the second extraction unit 102b generates LH1 by performing the following calculation from the values of the a-axis and b-axis of the Lab color space (step S203a-2).

ad = (a-ca) * cos (r) + b * sin (r) + ca
bd =-(a-ca) * sin (r) + b * cos (r)
LH1 = exp (-((ad * ad) / sa / sa + (bd * bd)
/ Sb / sb))

  Here, ad and bd are obtained by rotating the ab plane counterclockwise by r radian around (ca, 0). Also, as the value of r, about 0.3 to 0.8 radian is set. ca is set between 0 and 50. sa and sb are respectively the reciprocal of the sensitivity in the a-axis direction and the reciprocal of the sensitivity in the b-axis direction. Here, it is set as sa> sb.

Next, the second extraction means 102b limits the obtained LH1 with the luminance L. Those luminance L is set to 0 if the threshold value TH1 or more and LH 3 (Step S203a-3), the luminance L is the threshold TH2 following ones and LH3 (Step S203a-4). Here, the threshold TH1 is set between 60 and 100, and the threshold TH2 is set between 0 and 40. Let LH3 obtained here be a likelihood A indicating blood vessel likeness (step S203a-5).

  Description will be returned to FIG. The second extraction means 102b extracts blood vessel likeness as the likelihood A according to the above procedure (step S203a), and then multiplies the image BH after bottom hat processing by each element of the likelihood A indicating blood vessel likeness , Division by a factor N (step S204a). Furthermore, the blood vessel extraction image E enhanced by performing the clipping process in 1 is generated (step S205a).

According to the example described above, the blood vessel extraction image E, although a multi-level image having a value of up to 0-1, since the through ball Tomuhatto processing, the boundary of the extracted blood vessel is steeper. When it is desired to obtain a sharper boundary, binarization may be performed with a desired threshold.

  As described above, the second extracting unit 102b shifts the plane coordinates formed by the red hue direction of the color space and the blue hue direction around the specific point of the red hue axis. A likelihood A indicating the blood vessel likeness of the selected part is calculated by rotating the predetermined angle clockwise and limiting the luminance component in a specific value range. Then, the selected likelihood is emphasized by multiplying the calculated likelihood A by the luminance image obtained by performing bottom-hat processing on the image of the luminance component.

  A modification in which blood vessel likeness is extracted as the likelihood A will be described with reference to the flowchart in FIG. The extraction unit 102 acquires the value of the a-axis that is the red hue direction of the Lab color space (step S203a-7), and sets S to, for example, 80 (step S203a-8). A restriction is given in the range of ̃80 to perform normalization (A ← max (min (a, S), 0) / S), and the value is set in the range of 0 to 1 (step S 203 a-9). Here, the restriction is given with a value of 0 to 80, but this value is an example and is not limited to this value.

  Next, as another example of the image conversion process of a learning image, a method of directly extracting a blood vessel from color information will be described with reference to the flowcharts of FIGS. In the following description, a blood vessel likelihood image is generated from color information, and a blood vessel is extracted by the improved top hat processing (hereinafter referred to as morphology processing B). In the blood vessel likelihood image, the higher the likelihood, the larger the value of the image.

  In the morphological processing A shown in FIG. 6, the reduction processing is performed after the expansion processing is performed on the source image. The process of repeating expansion and contraction the same number of times is called closing process. That is, smoothing filtering is applied to the image subjected to closing processing, and subtraction (black hat processing) is performed from the source image. Here, the source image is a luminance image L, and the value of the image at the blood vessel is relatively small. Thus, when extracting a shape having a small value in the image, the morphological processing A shown in FIG. 6 is used.

  The blood vessel extraction process by the morphology process B will be described below. As shown in FIG. 9, the processing procedure of the image conversion B is as follows. First, the separation unit 101 performs Lab color space conversion on the photographed image in the RGB color space of the preprocessing unit 10 a (step S 201 b). Next, the pre-processing unit 10 a causes the second extraction unit 102 b to extract those textures (blood vessel likeness) of the selected part from the color information components separated in the Lab color space. Therefore, the second extraction unit 102b calculates the likelihood of a blood vessel as the likelihood A (step S202b). The method of determining the likelihood A is as described above using FIGS. 7 and 8.

  Subsequently, the second extraction unit 102b acquires a blood vessel extraction E image from the blood vessel likelihood image A indicating blood vessel likeness (step S203 b). A processing procedure for acquiring a blood vessel extraction image from the blood vessel likelihood image A is shown in FIG.

  According to the flowchart of FIG. 10, the second extraction unit 102b first performs contraction processing on the blood vessel likelihood image A using an appropriate structuring element to acquire the blood vessel likelihood image A1 after contraction processing (step S203 b). -1). Next, the blood vessel likelihood image A1 after the contraction process is subjected to dilation processing to obtain a blood vessel likelihood image A2 after the dilation processing (step S203 b-2). The second extraction means 102b further performs a smoothing filter process (Gaussian filter) on the blood vessel likelihood image A2 after the expansion process to obtain a blood vessel likelihood image A3 after the smoothing process (step S203 b-3). Finally, a blood vessel extraction image E is obtained by subtracting the blood vessel likelihood image A3 after the smoothing process from the blood vessel likelihood image A (step S203 b-4).

  As described above, the source image (blood vessel likelihood image A) subjected to contraction processing and then to expansion processing is referred to as opening processing. The second extraction unit 102b performs smoothing filter processing on the image subjected to opening processing, subtracts the opened image from the source image (top hat processing), and extracts the blood vessel shape in the source image. Here, since the source image is a blood vessel likelihood image, the value of the image is large where blood vessels are likely.

  The description is returned to the flowchart of FIG. After acquiring the blood vessel extraction E image from the blood vessel likelihood image A, the second extraction means 102b multiplies the blood vessel extraction E image by an appropriate coefficient N (step S204b), and is enhanced by performing the clipping process in 1. The blood vessel extraction image is generated (step S205 b).

  As described above, in order to acquire a shape having a large value in an image in order to acquire a shape from a multivalued image, a smoothing filter process is performed on the image subjected to the closing process, and the source image is subtracted. The blood vessel extraction image is obtained by obtaining the blood vessel extraction image, and on the other hand, when obtaining a shape having a small value in the image, the one subjected to the opening process is subjected to the smoothing filter process and subtracted from the source image. .

Here, the opening processing, refers to those repeated one or more times a contraction processing and expansion processing in that order, the claw-di-packaging process, repeated one or more times the expansion processing and contraction processing in that order Say what The shape of the structuring element used in both the opening process and the closing process is preferably a circle. Further, as a smoothing filter, for example, a Gaussian filter, an average filter, a median filter or the like is used.

  The above is the image conversion processing (blood vessel enhancement, clear structure) of the learning image, and the blood vessel shape extraction from the luminance image and the blood vessel shape extraction from the blood vessel likelihood image are performed. As a result, the irregular shape and the value fluctuation are large. It is possible to obtain the shape without generating a pseudo pattern such as moire for the shape.

  The description is returned to the flowchart of the machine learning discriminator generation process of FIG. In FIG. 4, after the preprocessing unit 10a executes the above-described image conversion processing for blood vessel enhancement and structure clarity (step S201), the conversion image is increased by eight times by a combination of 90 × N rotation and inversion. And deliver the result to the processing unit 11a (step S202).

In response to this, the processing unit 11a obtains 4096-dimensional feature vectors by averaging the learned CNN values of the respective increased images for each converted image (step S203). The processing unit 11a further concatenates the feature vector averages output for each transformed image to obtain a final vector representation (step S204). Then, stearyl-up S210 need types repeatedly performs the processes of S204 from discriminator generation after (step S205 "YES"), using the concatenated feature vectors, linear SVM (Support Vector Machine to enter the vector ) The learning is performed, and the machine learning discriminator generation process is ended (step S206).

  FIG. 11 is a flowchart showing the procedure of the test image identification process (step S30 of FIG. 3) of the diagnostic device 100A according to the first embodiment of the present invention.

  According to FIG. 11, first, the preprocessing unit 10a performs region enhancement and image conversion for clear structure with respect to unknown skin image data acquired by the acquisition unit 12a (step S301). The image conversion performed here follows the flowcharts of FIGS. 5 to 10 in the same manner as the image conversion of the learning skin image. Further, the preprocessing unit 10a executes an increase process eight times on each converted image by a combination of 90 × N rotation and inversion, and delivers the result to the processing unit 11a.

  The processing unit 11a obtains the 4096-dimensional feature vector by averaging the learned CNN values of the increased images for each converted image, and further concatenates the feature vector averages output for each converted image to obtain a final result. Vector representation is made (step S302). Then, after the processing unit 11a repeatedly executes the processing of steps S301 and S302 to generate the necessary type of classifier (step S303 "YES"), the determination unit 113 averages all the identification values and averages the final determination value. For example, it is displayed on the display device 120 (step S304).

  That is, in the processing of steps S301 to S303, the input image to be subjected to the image conversion is only changed from the learning skin image to the test image (an unknown image), and after the generation of the necessary types of classifiers The discrimination result is obtained by the output value of the linear SVM discriminator of

Second Embodiment
FIG. 12 shows the configuration of a diagnostic device 100B according to a second embodiment of the present invention. The difference in configuration from the diagnostic device 100A according to the first embodiment shown in FIG. 1 is only in the configuration of the processing unit 11b of the diagnostic device main body 10. In the diagnostic device 100B according to the second embodiment, the processing unit 11b is configured to generate a plurality of classifiers for each converted image including the original image. The rest of the configuration is the same as that of the diagnostic device 100A according to the first embodiment shown in FIG. 1 including the imaging device 110 with dermoscope as a peripheral device, the display device 120, and the input device 130.

  In the above configuration, the processing unit 11b identifies unit classifiers CNN1 to CNN3 (211 to 213) for the original image, classifiers CNN4 to CNN6 (214 to 216) for the site-emphasized image, and the structure clear image. The counters CNN7 to CNN9 (217 to 219) respectively generate feature vectors. In this case, the initial value at the time of learning at the time of classifier generation is determined by a random number each time, and different classifiers are made for the same converted image group.

  That is, in the diagnostic device 100B according to the second embodiment, each of the unit classifiers 211 to 219 is multistaged as a group of unit classifiers for each of the determined initial values by determining the initial value by the random number each time. Can function.

(Effect of the embodiment)
As described above, according to the diagnostic device 100A (100B) according to the present embodiment, after image conversion such as structural clarity or site enhancement is performed, image identification is individually performed for each conversion algorithm, and each identification score It is possible to improve the identification rate of disease by comprehensively determining

  Specifically, the discrimination rate by the warpage and the hokuro discrimination test is shown in the form of a table in FIG. 13 as the evaluation result. The target of identification was a 2-class classification of wart and hokuro. Wart can be caused by changes in the skin surface in areas with many viral infections and sebaceous glands, and uro is a cluster of cells called nevus cells in which the cells that make up the melanin pigment called melanocytes (melanocytes) are different. There are a certain number of cases where color and shape are similar and it is difficult for an amateur to distinguish. We chose this kind of problem setting in consideration of the direct demand for the identification of wart or hokuro and the fact that there are many cases with relatively easy access to data.

  The evaluation data was evaluated by dividing the warpage image and the black image into four parts using 261 war images and 704 black images. Furthermore, using this data set for evaluation, evaluation was performed using a discrimination / learning device of wart and hokuro. The identification success rate of the wart and the hokuro is shown in FIG. In the table, individual identification using training CNN (but without pre-processing), five image ensembles (concatenation of feature vectors by five types of converted images) and discrimination success rates by seven types of converted images are shown respectively. There is.

  As shown in FIG. 13, in the classifier using only individual converted images, 89.8% of gray images without pretreatment in the conventional BoF (Bag-of-Features) method, HDRC (High Dynamic Range Compression) gray The image was about 1 to 2% higher than 89.8%, and the superiority could be confirmed by the diagnostic device 100A (100B) according to the present embodiment. In addition, by using 7 types of ensemble method in which a structure clear image and a region-weighted image such as blood vessels are added to 5 types of ensemble methods, it is possible to confirm an improvement in the identification success rate from 93.9% to 94.7%. did it.

  Finally, although the diagnostic device, the learning processing method in the diagnostic device, and the program have been described above as the present embodiment, the diagnostic system may be constructed using each element as a means, for example, individual We connect means by network, place in medical field the means to acquire unknown skin image data related to the disease of the patient as a terminal, and specialize the means to identify the disease to be diagnosed based on the skin image data It may be arranged on a server owned by an organization.

  Although the present invention has been described above using the embodiment, it goes without saying that the technical scope of the present invention is not limited to the scope described in the above embodiment. It will be apparent to those skilled in the art that various changes or modifications can be added to the above embodiment. It is also apparent from the scope of the claims that the embodiments added with such alterations or improvements can be included in the technical scope of the present invention.

In the following, the invention described in the claims initially attached to the request for this application is appended. The item numbers of the claims described in the appendix are as in the claims attached at the beginning of the application for this application.
[Claim 1]
A diagnostic apparatus for diagnosing a skin disorder using a skin image, comprising:
An ensemble discriminator for identifying whether or not a disease is present based on a plurality of unknown skin image data related to a subject to be diagnosed;
The ensemble discriminator
In order to correspond to the plurality of skin image data including at least two of original image data relating to the object, first converted image data converted from the original image data, and second converted image data as well With two unit classifiers,
A determination unit that integrates identification values obtained separately by the unit classifier and obtains a final determination value.
[Claim 2]
The ensemble discriminator corresponds to the plurality of unknown skin image data including the original image data relating to the object, the first converted image data converted from the original image data, and the second converted image data. The diagnostic device according to claim 1, further comprising three unit discriminators.
[Claim 3]
The first image data converted from the original image data relating to the original image relating to the target, and the plurality of rotationally inverted images obtained by rotating and / or inverting the first image data converted by the ensemble discriminator The diagnostic device according to claim 1, further comprising at least two unit identifiers corresponding to the plurality of unknown skin image data including at least two of them.
[Claim 4]
Of the plurality of rotated and / or inverted images obtained by rotating and / or inverting the second converted image data converted from the original image data relating to the object, and the second converted image data by the ensemble discriminator The diagnostic device according to claim 1, further comprising at least two unit identifiers to correspond to the plurality of unknown skin image data including at least two.
[Claim 5]
The unit discriminator according to any one of claims 1 to 4, wherein each unit discriminator functions as a group of unit discriminators in multiple stages for each of the determined initial values, with the initial value being determined by a random number each time. The diagnostic device according to claim 1.
[Claim 6]
The diagnostic apparatus according to any one of claims 1 to 5, wherein the determination unit averages and integrates the plurality of identification values.
[Claim 7]
The first converted image data is image data obtained by subjecting the original image data to a structure clear conversion process, and the structure clear conversion process is a process of emphasizing a luminance component of the original image data. The diagnostic device according to any one of claims 1 to 6.
[Claim 8]
The second converted image data is image data obtained by performing site enhancement processing on the original image data, and the site enhancement conversion process is applied to the luminance component or the luminance component and the color information component of the original image data. The diagnostic device according to any one of claims 1 to 6, characterized in that the processing is for extracting.
[Claim 9]
The skin image database in which the identification name of the disease and the known skin image data are associated and recorded, and the acquiring unit capable of acquiring the plurality of unknown skin image data. The diagnostic device according to any one of 8.
[Claim 10]
The unit discriminator includes a convolutional neural network that learns based on a plurality of known skin image data related to a disease,
The convolutional neural network is
An input layer to which the plurality of known skin image data are input in the learning stage, and the plurality of unknown skin image data in the testing stage;
An intermediate layer having a plurality of sets composed of a convolution layer and a pooling layer and extracting features from the plurality of known skin image data or the plurality of unknown skin image data;
The diagnostic device according to any one of claims 1 to 9, further comprising: an output layer that outputs an identification value for each classification of a diagnostic object based on the extracted feature.
[Claim 11]
A learning processing method in a diagnostic apparatus comprising an ensemble discriminator for identifying whether or not a disease is present based on a plurality of unknown skin image data of a subject to be diagnosed,
Acquiring the plurality of unknown skin image data including at least two of original image data relating to the object, first converted image data converted from the original image data, and second converted image data as well;
Inputting each of the plurality of unknown skin image data into at least two unit classifiers constituting the ensemble classifier;
Integrating the identification values separately obtained by the unit discriminator to obtain a final judgment value.
[Claim 12]
A learning processing program of a diagnostic device comprising an ensemble discriminator for identifying whether or not a disease is present based on a plurality of unknown skin image data of a subject to be diagnosed,
On the computer
Acquiring a plurality of unknown skin image data including at least two of original image data relating to the object, first converted image data converted from the original image data, and second converted image data as well;
A step of inputting each of the plurality of unknown skin image data to at least two unit classifiers constituting the ensemble classifier;
A learning processing program for executing a procedure of integrating the identification values obtained separately by the unit classifier to obtain a final determination value.

DESCRIPTION OF SYMBOLS 10 Diagnostic device main body 10a, 10b: Pre-processing unit 11a, 11a: Processing unit (ensemble classifier) 12a, 12b: Acquisition unit 100A, 100B: Diagnostic device 103: Learning skin image storage unit 104 ... image storage unit, 103 ... skin image storage unit for learning, 110 ... imaging device with dermoscope, 120 ... display device, 130 ... input device, 101 ... separation means, 102 ... extraction means, 102a ... first extraction means, 102b ... second extraction means, 111, 112, 201-209 ... unit discriminator, 113 ... judgment means, 111 a ... input layer, 111 b ... intermediate layer, 111 c ... output layer, a ... processing unit, b ... feature map

Claims (19)

A diagnostic apparatus for diagnosing a skin disorder using a skin image, comprising:
An ensemble discriminator for identifying whether or not a disease is present based on a plurality of unknown skin image data related to a subject to be diagnosed;
The ensemble discriminator
Original image data according to the object, different from the first converted image data by the first conversion processing given on the original image data is performed, and the first conversion processing on the original image data of the second converted image data by the second conversion processing has been performed, so as to correspond to the plurality of unknown skin image data including the two, including at least the second converted image data, at least 2 With one unit identifier,
Determining means for integrating the identification values separately obtained by the unit classifier and obtaining a final determination value ;
The second conversion process is a process of emphasizing and extracting a portion corresponding to a blood vessel from the original image data based on a luminance component and a color information component of the original image data . The ensemble classifier, before Kihara image data, before SL so as to correspond to the first converted image data and the plurality of unknown skin image data including the second converted image data, the three unit identifier The diagnostic device according to claim 1, comprising: The ensemble classifier is added to the second converted image data, before Symbol least 2 of the first converted image data, and a plurality of rotary inversion image rotation and / or by inverting the first converted image data The diagnostic device according to claim 1, further comprising a plurality of unit identifiers corresponding to the plurality of unknown skin image data including one. The ensemble classifier, the addition to the second converted image data, before Symbol plurality of including at least two of the second converted image data rotated and / or inverted plurality of rotating and / or reverse image was a The diagnostic device according to claim 1, comprising a plurality of unit discriminators so as to correspond to unknown skin image data. The diagnostic apparatus according to any one of claims 1 to 4, wherein the determination unit averages and integrates the identification values obtained separately by the unit identifier . What the first structure clearly conversion process converts the image data to the original image data is image data that has undergone the structure clearly conversion processing emphasizing process der the luminance component of the original image data,
The ensemble discriminator includes a plurality of discriminators so as to correspond to the plurality of unknown skin image data including the first converted image data in addition to the second converted image data. The diagnostic device according to any one of claims 1 to 5. The skin image database in which the identification name of the disease and the known skin image data are associated and recorded, and the acquiring unit capable of acquiring the plurality of unknown skin image data. The diagnostic device according to any one of 6. The unit discriminator includes a convolutional neural network that learns based on a plurality of known skin image data related to a disease,
The convolutional neural network is
An input layer to which the plurality of known skin image data are input in the learning stage, and the plurality of unknown skin image data in the testing stage;
An intermediate layer having a plurality of sets composed of a convolution layer and a pooling layer and extracting features from the plurality of known skin image data or the plurality of unknown skin image data;
The diagnostic device according to any one of claims 1 to 7 , further comprising: an output layer that outputs an identification value for each classification of a diagnostic object based on the extracted feature . A learning processing method in a diagnostic apparatus comprising an ensemble discriminator for identifying whether or not a disease is present based on a plurality of unknown skin image data of a subject to be diagnosed,
The original image data relating to the target, the first converted image data obtained by performing the predetermined first conversion processing on the original image data, and the first conversion processing on the original image data are different from each other Acquiring the plurality of unknown skin image data including at least two of the second conversion image data subjected to the second conversion processing including the second conversion image data;
Inputting each of the plurality of unknown skin image data into at least two unit classifiers constituting the ensemble classifier;
Combining a plurality of identification values separately obtained by the unit identifier to obtain a final determination value;
The second conversion process, the original on the basis of the luminance component and color information components of the image data, diagnostics you, wherein a process of extracting highlights the portion corresponding to the blood vessel from the original image data Learning processing method in the device. It is a program of a diagnostic device provided with an ensemble discriminator which identifies whether it is a disease based on a plurality of unknown skin image data concerning a subject to be diagnosed,
In the diagnostic device,
The original image data relating to the target, the first converted image data obtained by performing the predetermined first conversion processing on the original image data, and the first conversion processing on the original image data are different from each other A step of acquiring the plurality of unknown skin image data including two of the second conversion image data subjected to the second conversion processing including at least the second conversion image data;
A step of inputting each of the plurality of unknown skin image data to at least two unit classifiers constituting the ensemble classifier;
Performing a procedure of combining a plurality of identification values obtained separately by the unit identifier to obtain a final determination value;
The second conversion process is a program for emphasizing and extracting a portion corresponding to a blood vessel from the original image data based on a luminance component and a color information component of the original image data. The second conversion process includes a predetermined morphological process performed on a luminance image corresponding to the luminance component of the original image data to obtain the second converted image data. A diagnostic device according to any one of the preceding claims. 9. The method according to claim 1, wherein the second conversion process includes a process of obtaining a likelihood indicating the blood vessel likeness based on the color information component in order to obtain the second converted image data. The diagnostic device according to any one of 11. The second conversion processing is characterized in that the second conversion image data is generated based on the luminance image on which the morphological processing has been performed and the likelihood indicating the blood vessel likeness thus obtained. The diagnostic device according to claim 12, which is subordinate to claim 11. The second conversion process includes a predetermined morphological process performed on a luminance image corresponding to the luminance component of the original image data to obtain the second converted image data. The learning processing method in the diagnostic device according to claim 9. The method according to claim 9 or 14, wherein the second conversion process includes a process of obtaining a likelihood indicating the blood vessel likeness based on the color information component in order to obtain the second converted image data. The learning processing method in the diagnostic apparatus as described. The second conversion processing is characterized in that the second conversion image data is generated based on the luminance image on which the morphological processing has been performed and the likelihood indicating the blood vessel likeness thus obtained. The learning processing method in the diagnostic device according to claim 15, which is subordinate to claim 14. The second conversion process includes a predetermined morphological process performed on a luminance image corresponding to the luminance component of the original image data to obtain the second converted image data. The program according to claim 10. The method according to claim 10, wherein the second conversion process includes a process of obtaining a likelihood indicating the blood vessel likeness based on the color information component in order to obtain the second converted image data. Described program. The second conversion processing is characterized in that the second conversion image data is generated based on the luminance image on which the morphological processing has been performed and the likelihood indicating the blood vessel likeness thus obtained. The program according to claim 18, which depends on claim 17.

Images