JP6987721B2 - Image processing equipment, methods and programs - Google Patents

Description

The present disclosure relates to an image processing apparatus, method and program for classifying each pixel included in a cross-sectional image into a plurality of types of findings.

In recent years, with the progress of medical devices such as CT (Computed Tomography) devices and MRI (Magnetic Resonance Imaging) devices, higher quality and higher resolution three-dimensional images have come to be used for diagnostic imaging.

By the way, interstitial pneumonia is known as a lung disease. By analyzing the cross-sectional image of a patient with interstitial pneumonia, the lesion or tissue showing specific symptoms such as honeycomb lung, reticular shadow and cyst included in the cross-sectional image (hereinafter, the lesion or tissue is collectively referred to as a finding). ) Has been proposed as a method for classifying and quantifying. By analyzing the cross-sectional image in this way to classify and quantify the findings, the degree of lung disease can be easily determined. Further, by assigning a different color to each of the findings areas classified in this way for each finding, it is possible to easily diagnose how much a specific finding area is included in the image.

Japanese Unexamined Patent Publication No. 2015-80720

However, in the conventional method, since the two-dimensional cross-sectional image is classified, it may be classified as a discontinuous region with respect to the depth direction of the two-dimensional cross-sectional image. Such a classification is not an anatomically appropriate classification.

In Patent Document 1, in order to determine whether the lesion included in the findings is benign or malignant, a plurality of two-dimensional images are created from the three-dimensional images, and the lesions are evaluated and determined in each of the plurality of two-dimensional images. A method has been proposed. However, Patent Document 1 only classifies whether the lesion contained in the two-dimensional image is benign or malignant, and does not propose any method for appropriately classifying the lesion region.

In view of the above circumstances, it is an object of the present disclosure to provide an image processing apparatus, method and program capable of classifying each pixel of a cross-sectional image into a plurality of types of findings with high accuracy.

The image processing apparatus according to the present disclosure includes a cross-sectional image acquisition unit that acquires cross-sectional images of a plurality of different cross-sectional directions of a subject.
For each of the plurality of cross-sectional images, a primary classification unit that performs a primary classification process for specifying the type of finding in which each pixel of each cross-sectional image is classified, and
For the parts common to multiple cross-sectional images, the parts common to multiple cross-sectional images are classified by evaluating the results of the primary classification processing of each cross-sectional image by different evaluation methods according to the type of findings. It is provided with a secondary classification unit that performs secondary classification processing to specify the type.

The "part common to a plurality of cross-sectional images" may be one common pixel as long as it is common to a plurality of cross-sectional images, and a plurality of a plurality within a predetermined range around the common pixel. It may be a pixel of.

In the image processing apparatus according to the present disclosure, the result of the primary classification process may include an evaluation value indicating that each of the plurality of types of findings is used.

Further, in the image processing apparatus according to the present disclosure, the secondary classification unit preferentially classifies the findings having higher evaluation values into the findings of the common part as the evaluation value in all the cross-sectional images is higher. The conversion method may be adopted as one of the evaluation methods.

Further, in the image processing apparatus according to the present disclosure, the specific averaging method may be a harmonic mean.

Further, in the image processing apparatus according to the present disclosure, the secondary classification unit may perform secondary classification processing on the evaluation value of the lesion findings by a specific averaging method.

Further, in the image processing apparatus according to the present disclosure, the secondary classification unit performs secondary classification processing on the evaluation values for findings other than lesions by another evaluation method different from the specific averaging method. There may be.

Further, in the image processing apparatus according to the present disclosure, when the findings include the boundary between lesions and the boundary between the lesion and the background tissue other than the lesion, the secondary classification unit determines the evaluation value of the boundary findings. , The secondary classification process may be performed by a specific averaging method.

Further, in the image processing apparatus according to the present disclosure, when the findings include the target organ, the secondary classification unit performs the secondary classification process on the evaluation value of the findings of the target organ by a specific averaging method. May be.

Further, in the image processing apparatus according to the present disclosure, the secondary classification unit performs secondary classification processing on the evaluation values for findings other than the target organ by another evaluation method different from the specific averaging method. May be.

In this case, the other evaluation method may be an evaluation method in which the representative value of the evaluation value in the portion common to the plurality of cross-sectional images is used as the result of the secondary classification process.

Further, in the image processing apparatus according to the present disclosure, the secondary classification unit uses the findings obtained with the highest evaluation for the portion common to a plurality of cross-sectional images as the findings for the pixels as a result of the secondary classification processing. It may be specific.

Further, in the image processing apparatus according to the present disclosure, the primary classification unit may perform primary classification processing using a discriminator generated by machine learning.

Further, the image processing apparatus according to the present disclosure may further include a display control unit that displays the result of the secondary classification process on the display unit.

In this case, the display control unit may further display the result of the primary classification process on the display unit.

Further, the image processing apparatus according to the present disclosure may further include a correction unit that corrects the result of the secondary classification processing based on the anatomical features of the findings or the amount of image features.

Further, in the image processing apparatus according to the present disclosure, the correction unit may correct the result of the secondary classification process based on the signal value of each pixel of the cross-sectional image in a specific cross-sectional direction.

Further, in the image processing apparatus according to the present disclosure, the correction unit may correct the result of the secondary classification process based on the anatomical features of the shape or position of the tissue or lesion.

Further, in the image processing apparatus according to the present disclosure, the correction unit has an abnormality in the continuity of the edge or center of gravity of the tissue or lesion specified by each pixel classified by the secondary classification process in the three-dimensional space. In some cases, the result of the secondary classification process of each pixel may be corrected.

The image processing method according to the present disclosure obtains cross-sectional images of a plurality of different cross-sectional directions of a subject, and obtains cross-sectional images.
For each of the plurality of cross-sectional images, a primary classification process is performed to specify the type of finding in which each pixel of each cross-sectional image is classified.
For the parts common to multiple cross-sectional images, the parts common to multiple cross-sectional images are classified by evaluating the results of the primary classification processing of each cross-sectional image by different evaluation methods according to the type of findings. Performs secondary classification processing to specify the type.

It should be noted that the image processing method according to the present disclosure may be provided as a program for causing a computer to execute the image processing method.

Other image processing devices according to the present disclosure include a memory for storing instructions for causing a computer to execute, and a memory.
The processor comprises a processor configured to execute a stored instruction.
Obtain cross-sectional images of multiple different cross-sectional directions of the subject,
For each of the plurality of cross-sectional images, a primary classification process is performed to specify the type of finding in which each pixel of each cross-sectional image is classified.
For the parts common to multiple cross-sectional images, the parts common to multiple cross-sectional images are classified by evaluating the results of the primary classification processing of each cross-sectional image by different evaluation methods according to the type of findings. Performs secondary classification processing to specify the type.

According to the present disclosure, each pixel of a cross-sectional image can be classified into a plurality of types of findings with high accuracy.

A block diagram showing a schematic configuration of a medical image diagnosis support system using an embodiment of the image processing apparatus of the present disclosure. The figure which shows an example of the sectional image in a plurality of different sectional directions. Diagram showing an example of a multi-layer neural network A diagram showing evaluation values corresponding to the types of findings for the central pixel of a region of interest. Diagram for explaining the secondary classification process A diagram showing the arithmetic mean of the evaluation values of the findings for the central pixel of a region of interest. A diagram showing the harmonic mean of the evaluation values of the findings for the central pixel of a region of interest. A diagram showing the results of secondary classification when different evaluation methods are adopted according to the type of findings. Diagram showing an example of a mapping image A flowchart for explaining the operation of the medical image diagnosis support system using one embodiment of the image processing apparatus of the present disclosure. Diagram to illustrate near the boundaries of the region A block diagram showing a schematic configuration of a medical image diagnosis support system using another embodiment of the image processing apparatus of the present disclosure. Diagram showing an example of misclassification of cysts Diagram showing other examples of misclassified cysts Diagram to explain the continuity of the position of the center of gravity of each region classified as a cyst

Hereinafter, a medical image diagnosis support system using an embodiment of the image processing apparatus, method, and program of the present disclosure will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing a schematic configuration of a medical image diagnosis support system using an embodiment of the image processing apparatus of the present disclosure.

As shown in FIG. 1, the medical image diagnosis support system of the present embodiment includes an image processing device 10, a display device 20, an input device 30, and a three-dimensional image storage server 40.

The image processing device 10 is configured by installing the image processing program of the present embodiment on a computer. The image processing program is recorded and distributed on a recording medium such as a DVD (Digital Versatile Disc) or a CD-ROM (Compact Disk Read Only Memory), and is installed in a computer from the recording medium. Alternatively, it is stored in a storage device of a server computer connected to a network or in a network storage in a state accessible from the outside, and is downloaded and installed on a computer used by a doctor upon request.

The image processing device 10 includes a CPU (Central Processing Unit), a semiconductor memory, a hard disk, a storage device such as an SSD (Solid State Drive), and the like. Then, these hardware constitute a cross-sectional image acquisition unit 11, a primary classification unit 12, a secondary classification unit 13, and a display control unit 14 as shown in FIG. Then, each of the above parts operates by executing the image processing program installed in the storage device by the CPU.

The cross-sectional image acquisition unit 11 acquires a cross-sectional image of a subject taken in advance before surgery, examination, or the like. The cross-sectional image may be represented by, for example, slice data output from a CT device, an MRI device, or the like, slice data output from an MS (Multi Slice) CT device, a cone beam CT device, or the like. The cross-sectional image is stored in the three-dimensional image storage server 40 in association with the identification information of the subject. The cross-sectional image acquisition unit 11 reads out the cross-sectional image corresponding to the identification information of the subject input in the input device 30 from the three-dimensional image storage server 40 and acquires it.

Further, the cross-sectional image acquisition unit 11 acquires a plurality of cross-sectional images of the subject in different cross-sectional directions. The cross-sectional direction includes axial, sagittal, coronal, oblique and the like. It is preferable that the cross-sectional image acquisition unit 11 acquires at least three or more cross-sectional images. As shown in FIGS. A, B, and C of FIG. 2, the cross-sectional image acquisition unit 11 acquires a plurality of cross-sectional images for each of the three different cross-sectional directions. The three cross-sectional directions are the axial direction, the sagittal direction, and the coronal direction.

When the cross-sectional image acquisition unit 11 acquires a plurality of cross-sectional images of the subject in different cross-sectional directions in advance in the three-dimensional image storage server 40, the cross-sectional image acquisition unit 11 acquires the plurality of cross-sectional images of the subject. When the volume data is stored in the three-dimensional image storage server 40, the volume data may be read out, and a plurality of cross-sectional images in different cross-sectional directions may be generated and acquired from the volume data.

The primary classification unit 12 is a primary classification for specifying the type of tissue or lesion to which each pixel of each cross-sectional image is classified, that is, the type of findings, for each of the plurality of cross-sectional images acquired by the cross-sectional image acquisition unit 11. Perform processing. In the present embodiment, a plurality of cross-sectional images including the lung region are acquired by the cross-sectional image acquisition unit 11, and evaluation values indicating that each pixel of each cross-sectional image is each of a plurality of types of findings, that is, a plurality of types. The evaluation values corresponding to each of the findings are acquired as the result of the primary classification process. Hereinafter, among the plurality of cross-sectional images, only the primary classification process for one cross-sectional image in the axial direction will be described, but it is assumed that the primary classification process is also performed for the cross-sectional images in the other cross-sectional directions.

The primary classification unit 12 of the present embodiment has a discriminator composed of a multi-layer neural network generated by deep learning (deep learning) which is machine learning, and the discriminator is used for each pixel of each cross-sectional image. Obtain the evaluation value corresponding to each of multiple types of findings.

In the multi-layer neural network, in each layer, arithmetic processing is performed using various kernels on the data of a plurality of different features obtained by the layer of the previous stage. Then, by further performing arithmetic processing on the feature quantity data obtained by this arithmetic processing in the next and subsequent layers, the recognition rate of the feature quantity is improved, and the input data corresponds to each of a plurality of classes. It is possible to acquire the evaluation value to be used and classify the input data based on the evaluation value into one of a plurality of classes.

FIG. 3 is a diagram showing an example of a multi-layer neural network. As shown in FIG. 3, the multi-layer neural network 50 includes a plurality of layers including an input layer 51 and an output layer 52. In the present embodiment, the lung region included in the cross-sectional image is, for example, an infiltrative shadow, a mass shadow, a ground glass shadow, a lobular central nodule shadow, a non-lobular central nodule shadow, a reticular shadow, a linear shadow, and a thickening of the mediastinal wall. , Honeycombing, cyst, low absorption area (emphysema), cavity, pleural thickening, pleural effusion, cavity, bronchiectal dilation, traction bronchiectal dilation, arteries, normal lungs, chest wall and mediastinum. It has been done. The types of findings are not limited to these, and may be more or less than these.

In this embodiment, these findings are trained by the multi-layer neural network 50 using a large number of teacher data of millions. At the time of learning, a region of interest having a predetermined size (for example, 1.5 cm × 1.5 cm) is cut out from a cross-sectional image of which the type of finding is known, and the region of interest is used as teacher data. Then, the teacher data is input to the multi-layer neural network 50, and the result of the classification process of the type of findings (hereinafter referred to as the classification result) is output. Next, the output result is compared with the teacher data, and the units included in each layer of the multi-layer neural network 50 from the output side to the input side depending on whether the answer is correct or incorrect (circled in FIG. 3). Correct the join weight between each layer of. The correction of the join weight is repeated using a large number of teacher data until the correct answer rate of the predetermined number of times or the output classification result reaches 100%, and the learning is completed.

When the primary classification unit 12 performs the primary classification process, the region of interest having the same size as the teacher data is sequentially cut out from the cross-sectional image of the classification target, and the region of interest is input to the discriminator composed of the multilayer neural network 50. As a result, the evaluation values corresponding to each of the plurality of findings are output for the central pixel of the cut out region of interest. The evaluation value corresponding to each of the plurality of findings is an evaluation value indicating the possibility that the central pixel belongs to each finding, and the larger the evaluation value, the higher the possibility of being classified into the findings. means. The primary classification unit 12 shall acquire evaluation values corresponding to each of the plurality of types of findings in each pixel included in the cross-sectional image.

FIG. 4 is a diagram showing evaluation values corresponding to the types of findings for the central pixel of a certain region of interest. Note that FIG. 4 shows evaluation values for some of the plurality of findings for the sake of simplicity. In the present embodiment, the discriminator outputs an evaluation value corresponding to each of the plurality of findings for the central pixel of the region of interest. Therefore, as shown in FIG. 4, the evaluation values corresponding to each finding are acquired. When the evaluation value as shown in FIG. 4 is acquired, the central pixel of the region of interest is most likely to be a reticular shadow, and then is most likely to be a ground glass shadow. On the contrary, there is almost no possibility of normal lung or low absorption area. Therefore, if only the primary classification result is used, the central pixel of the region of interest from which the evaluation value is acquired as shown in FIG. 4 is classified into the reticulated shadow having the maximum evaluation value.

As described above, the primary classification unit 12 performs the same primary classification processing on the cross-sectional images in the cross-sectional directions other than the cross-sectional images in the axial direction as described above, and in each pixel of each cross-sectional image, a plurality of types of findings are found. Obtain the evaluation value corresponding to each.

Next, the secondary classification unit 13 will be described. The secondary classification unit 13 evaluates the primary classification result of each cross-sectional image by a different evaluation method according to the type of findings for the portion common to the plurality of cross-sectional images used in the primary classification unit 12. In particular, for the portion common to a plurality of cross-sectional images, the primary classification result of each cross-sectional image is evaluated by a different evaluation method according to the type of findings. As a result, the secondary classification unit 13 identifies the type of finding in which the pixels common to the plurality of cross-sectional images are classified. Specifically, as shown in FIG. 5, the primary classification results of the cross-sectional images S1, S2, and S3 are acquired for the pixels P common to the cross-sectional images S1, S2, and S3 in the three directions, and the three primary classifications are obtained. The results are evaluated to identify the type of finding to which the pixel P is classified.

In the present embodiment, as a result of the primary classification, evaluation values corresponding to each of the plurality of types of findings are acquired in each pixel of the cross-sectional image. Therefore, as an evaluation method for the primary classification result, it is conceivable to use an evaluation method for calculating the arithmetic mean value of the evaluation values for each finding. In this case, the finding for which the largest arithmetic mean value is calculated is specified as the final finding of the pixel P.

FIG. 6 is a diagram showing an arithmetic mean value of evaluation values of findings for a central pixel of a certain region of interest. As shown in FIG. 6, the arithmetic mean value of the evaluation value of the findings for this pixel has the largest infiltration shadow of 6.3. Therefore, when the arithmetic mean is used as the evaluation method, this pixel is classified as an infiltration shadow by the secondary classification process. However, when comparing the evaluation values of the cross sections in the three directions of coronal, sagittal, and axial for the infiltration shadow, the evaluation value in the axial direction is 0.5, and the evaluation value in the coronal direction is 10.8 and the sagittal direction. It is lower than the evaluation value of 7.5. The evaluation value of 0.5 indicates that the pixel has an extremely low possibility of infiltration shadow in the cross-sectional image in the axial direction. Therefore, when the pixel is finally classified as an infiltration shadow in such a situation, the classification accuracy is not always high.

In general, since lesions have many convex structures, in the lesions included in the learning data for learning the multi-layer neural network 50, the periphery of the lesion is often the tissue that is the background of the normal lung or the like. When an attempt is made to learn a multi-layer neural network 50 so that a lesion can be reliably detected using such learning data, the lesion is learned more strongly than the background such as a normal lung. As a result, when the multi-layer neural network 50 learned in this way is used as a discriminator, a region where the classification of a lesion or a background is delicate, such as the boundary of a lesion, is easily classified as a lesion widely. On the contrary, a wide area such as a normal lung, which is the background, is difficult to be classified for a lesion.

Further, even in a region other than the lesion region, there are few states in which no lesion exists in a certain cross-sectional image as in the evaluation value of the cross-sectional image in the axial direction of the infiltrating shadow shown in FIG. 6, and the multi-layer neural network is actually used. The evaluation value output by the network 50 often varies slightly depending on the type of finding.

Therefore, in order to make the difference larger than the classification result, it is preferable to adopt a different evaluation method depending on the findings of the lesion and other findings such as other backgrounds.

Therefore, in the present embodiment, the secondary classification unit 13 has a higher evaluation value for the findings of the lesion based on the primary classification result, as the evaluation value in all the cross-sectional images is higher. A specific averaging method is adopted that preferentially classifies the findings into the findings of common pixels. Specifically, the harmonic mean is adopted as the evaluation method as a specific averaging method. In this case, the above-mentioned arithmetic mean is adopted as the evaluation method for the evaluation values for the findings other than the lesion. Here, the harmonic mean is expressed by the following equation (1). In the formula (1), E1 is an evaluation value in the cross-sectional image in the coronal direction, E2 is an evaluation value in the cross-sectional image in the sagittal direction, and E3 is an evaluation value in the cross-sectional image in the axial direction. Further, the secondary classification unit 13 calculates the harmonic mean only for the findings whose evaluation value exceeds 0, and does not calculate the harmonic mean for the findings whose evaluation value is 0 or less.

Harmonic mean = 3 / (1 / E1 + 1 / E2 + 1 / E3) (1)

FIG. 7 is a diagram showing a harmonic mean value of evaluation values for each finding for a certain pixel similar to that in FIG. As shown in FIG. 7, when the harmonic mean is adopted as the evaluation method, the harmonic mean value becomes a low value when a low evaluation value such as an infiltration shadow is included. Conversely, for findings with large evaluation values in all three directions, such as normal lung, the harmonic mean is large. When the harmonic mean is used as the evaluation method as shown in FIG. 7, the secondary classification result which is the result of the secondary classification process of the pixel is the normal lung having the largest harmonic mean value.

As mentioned above, it is preferable to use the harmonic mean for the evaluation value of the lesion findings. However, in a relatively small region such as a punctate shadow, it is difficult for the positions of the pixels classified as punctate shadows to match in the cross-sectional image in three directions. For this reason, the positions of the pixels that should be punctate shadows in the cross-sectional image in three directions are only slightly off, and the pixels that should be punctate shadows on one tomographic plane may not be recognized at all on other tomographic planes. be. In such a case, if the harmonic mean is adopted as the evaluation method, the pixels will not be classified as point-like shadows. For this reason, it is preferable to use an arithmetic mean instead of a harmonic mean as the evaluation method for the evaluation value of the finding that becomes a punctate shadow.

Further, a linear structure such as an artery, that is, a blood vessel is included linearly in a cross-sectional image in a certain direction, but is included as a point in a cross-sectional image in a direction orthogonal to this direction. For this reason, the positions of the pixels that should have a linear structure in the cross-sectional image in three directions are only slightly deviated, and the pixels that should have a linear structure in one tomographic plane may not be recognized at all in another tomographic plane. be. In such a case, if the harmonic mean is adopted as the evaluation method, the pixels will not be classified into linear structures such as blood vessels. For this reason, it is preferable to use an arithmetic mean instead of a harmonic mean as the evaluation method for the evaluation values of the findings having a linear structure.

In addition, lesions such as honeycomb lungs look like a honeycomb viewed from above in a cross-sectional image in the axial direction, but are often accompanied by a vertically long structure in a cross-sectional image in the coronal direction or the sagittal direction. On the other hand, in diagnostic imaging using CT images or MRI images, cross-sectional images in the axial direction are often used. Therefore, the learning data used for learning the multi-layer neural network 50 also has many cross-sectional images in the axial direction. When the multi-layer neural network 50 learned in this way is used as a discriminator, the accuracy of classification of the honeycomb lung is high for the sectional image in the axial direction, but the sectional image in the coronal direction and the sectional image in the sagittal direction are classified into the honeycomb. The classification accuracy of the lungs is lower than that of the cross-sectional image in the axial direction. In such a case, a weighted arithmetic mean in which the weight of the evaluation value for the cross-sectional image in the axial direction is larger than the weight of the evaluation value for the cross-sectional image in the coronal direction and the cross-sectional image in the sagittal direction is adopted as the evaluation method. , Honeycomb lung classification accuracy can be improved. Further, the evaluation method in such a case is not limited to the weighted arithmetic mean. Depending on the type of lesion, representative values other than the weighted arithmetic mean, such as the median and maximum values for the evaluation values of the cross-sectional images in three directions, may be used.

As described above, in the present embodiment, different evaluation methods are adopted according to the types of findings.

FIG. 8 is a diagram showing the results of secondary classification when different evaluation methods are adopted according to the type of findings. In FIG. 8, the “double weight of the axial” of the evaluation method of the honeycomb lung is the evaluation of the evaluation value for the cross-sectional image in the axial direction, the evaluation value for the cross-sectional image in the coronal direction, and the evaluation for the cross-sectional image in the sagittal direction. It means a weighted arithmetic average that is twice the value. Further, in FIG. 8, the maximum value of the evaluation method for normal lung means an evaluation method that adopts the maximum value among the evaluation values in three directions. As shown in FIG. 8, when different evaluation methods are adopted depending on the type of findings, the secondary classification result, which is the result of the secondary classification processing of this pixel, is the honeycomb lung having the largest evaluation value of 3.9. .. By adopting different evaluation methods according to the types of findings in this way, classification can be performed more accurately.

In the above description, the secondary classification process of one pixel P has been described, but it is found that all the pixels in the three-dimensional space are subjected to the secondary classification process in the same manner as described above, and each pixel is classified. The type of is specified.

Returning to FIG. 1, the display control unit 14 generates a mapping image by assigning a color to each classification area based on the secondary classification result, and displays the mapping image on the display device 20.

Specifically, the display control unit 14 applies the same color to all the pixels in the three-dimensional space classified into any of the above-mentioned plurality of types of findings with respect to the pixels classified into the same findings. By assigning, a three-dimensional mapping image is generated. FIG. 9 is a diagram showing an example of a mapping image. Note that FIG. 9 shows mapping images classified into eight types of findings: ground glass shadow, normal lung, bronchiectasis, honeycomb lung, reticular shadow, infiltration shadow, and cyst for the sake of simplicity. .. Further, although FIG. 9 shows a cross-sectional image of an arbitrary cross section in the three-dimensional mapping image, the present invention is not limited to this, and the three-dimensional mapping image may be displayed on the display device 20.

Further, the display control unit 14 displays a plurality of cross-sectional images in the cross-sectional direction acquired by the cross-sectional image acquisition unit 11 and a cross-sectional image or a three-dimensional image subjected to the secondary classification process by the secondary classification unit 13 on the display device 20. It can be displayed. The display control unit 14 classifies each pixel of the cross-sectional image into the finding that maximizes the evaluation value based on the primary classification result by the primary classification unit 12, and generates a mapping image based on the primary classification result. There may be. In this case, the mapping image shown in FIG. 9 (that is, the mapping image based on the secondary classification result) and the mapping image based on the primary classification result may be switchable, or these may be displayed side by side. This makes it possible to easily compare the primary classification result with the secondary classification result.

The display device 20 includes, for example, a liquid crystal display or the like. Further, the display device 20 may be configured by a touch panel and may also be used as the input device 30.

The input device 30 includes a mouse, a keyboard, and the like, and accepts various setting inputs by the user. Specifically, for example, it accepts setting input of patient identification information.

Next, the operation of the image processing apparatus of this embodiment will be described. FIG. 10 is a flowchart showing the operation of the image processing apparatus of the present embodiment. First, a cross-sectional image in a plurality of cross-sectional directions of the patient is acquired in response to a setting input such as patient identification information by the user (step ST1). The plurality of cross-sectional images are output to the primary classification unit 12, and the primary classification unit 12 acquires the primary classification result by performing the primary classification process on each of the input cross-sectional images (step ST2).

Then, the primary classification result by the primary classification unit 12 is output to the secondary classification unit 13, and the secondary classification unit 13 uses a different evaluation method according to the type of findings based on the input primary classification result to perform secondary classification. Perform processing (step ST3).

Then, the secondary classification result by the secondary classification unit 13 is output to the display control unit 14, and the display control unit 14 generates a mapping image by assigning colors to the areas of each classification, and the mapping image is displayed on the display device 20. Is displayed (step ST4), and the process is terminated.

As described above, according to the present embodiment, a plurality of cross-sectional images of the subject in different cross-sectional directions are acquired, and for each of the plurality of cross-sectional images, the type of finding in which each pixel of each cross-sectional image is classified is specified. Perform the primary classification process for this purpose. Next, with respect to the part common to the plurality of cross-sectional images, the part common to the plurality of cross-sectional images is classified by evaluating the primary classification result of each cross-sectional image by a different evaluation method according to the type of the finding. Performs secondary classification processing to specify the type. By performing the secondary classification process in this way, each pixel of the cross-sectional image can be classified into a plurality of types of findings with high accuracy.

In the above embodiment, a plurality of types of findings may include boundaries between lesions. Here, the boundary between lesions means a predetermined range including the boundary between lesions (for example, several pixels around the boundary). Specifically, as shown in FIG. 11, it is assumed that the lesion areas A1 and A2 are adjacent to each other. The vicinity of the boundary of the area A1 is the area A11 in a predetermined range from the boundary B0 between the area A1 and the area A2. On the other hand, the vicinity of the boundary of the area A2 is the area A12 in a predetermined range from the boundary B0 between the area A1 and the area A2. It should be noted that the plurality of types of findings may include the boundary between the lesion and the tissue other than the lesion.

When boundaries such as between lesions are included in multiple types of findings in this way, the higher the evaluation value in all cross-sectional images, such as the harmonic mean, for the evaluation value of the boundary obtained by the primary classification result. It is preferable to adopt an evaluation method that preferentially classifies the findings with high evaluation values into the findings of the common part. In this case, the above-mentioned arithmetic mean may be adopted as the evaluation method for the evaluation values other than the boundary.

Further, in the above embodiment, the target organs such as the lung and the liver to be diagnosed may be included as the findings. In this way, when the target organ is included in the findings, the higher the evaluation value in all the cross-sectional images, the higher the evaluation value, as in the harmonic mean, for the evaluation value of the target organ obtained from the primary classification result. It is preferable to adopt an evaluation method that preferentially classifies the findings that have become common to the findings of the common part. In this case, for the evaluation values other than the target organ, the above-mentioned arithmetic mean may be adopted as the evaluation method.

Further, in the above embodiment, the secondary classification result output by the secondary classification unit 13 may be corrected based on the anatomical features of the findings or the amount of image features. Hereinafter, this will be described as another embodiment. FIG. 12 is a block diagram showing a schematic configuration of a medical image diagnosis support system using the image processing apparatus and method of the present disclosure and other embodiments of the program. In FIG. 12, the same reference numbers are assigned to the same configurations as those in FIG. 1, and detailed description thereof will be omitted. The medical image diagnosis support system according to the other embodiment shown in FIG. 12 is provided with a correction unit 15 for the medical image diagnosis support system shown in FIG.

The correction unit 15 corrects the secondary classification result based on the anatomical feature of the finding or the image feature amount.

Here, even if the finding that each pixel is classified by the above-mentioned secondary classification unit 13 is specified, the classification result may be different from the result according to the anatomical structure.

Specifically, for example, the contour of a cyst may be a continuous smooth curve, but as a result of secondary classification, the contour of a region consisting of pixels classified as a cyst may not be a continuous smooth curve. For example, as shown in FIG. 13, a region composed of pixels classified as a cyst may be two divided regions r1 and r2. Further, as shown in FIG. 14, there are cases where a protrusion is formed on the contour of the region r3 composed of pixels classified as a cyst, or a dent is formed on the contrary, and the edge of the region of the cyst is abnormal.

Therefore, the correction unit 15 of the other embodiment of the present disclosure corrects the secondary classification result so that the result is in line with the anatomical structure. Specifically, the correction unit 15 corrects the secondary classification result of each pixel by using a cross-sectional image in a specific cross-sectional direction. As the cross-sectional image in a specific cross-sectional direction, it is preferable to use a cross-sectional image capable of classifying each pixel into findings with higher accuracy. It is preferable to use a cross-sectional image in a high cross-sectional direction.

Specifically, for example, the cyst region is extracted by performing threshold processing or filter processing using the pixel value of the cross-sectional image in the cross-sectional direction having the highest resolution, and the contour thereof is acquired as an image feature amount. Then, the correction is performed by matching the region consisting of the pixels classified as the cyst by the secondary classification process with the contour. Thereby, the region r2 shown in FIG. 13 can be excluded from the region of the cyst, and the protrusion of the region r3 shown in FIG. 14 can be eliminated. Further, not only the cyst but also the bronchial region is extracted by performing threshold processing or filter processing using the pixel value of the cross-sectional image, and the contour thereof is acquired as an image feature amount. Then, the correction may be performed by matching the region consisting of the pixels classified as the bronchi by the secondary classification process with the contour.

In the above description, the cyst region or the bronchial region is extracted, but the findings do not necessarily have to be specified. For example, the edge or shape contained in the cross-section image may be analyzed to segment the cross-section image, and the region consisting of pixels classified into specific findings by the secondary classification process may be expanded or narrowed to the segmented region. good.

Further, when making corrections, the signal values included in the area classified by the secondary classification process may be used as the image feature amount. Specifically, the distribution of signal values included in the regions classified by the secondary classification process is analyzed, and if there are high signal value regions and low signal value regions, these regions are used using a discriminant analysis method. By separating the above-mentioned regions of signal values representing tissues such as cysts or bronchi as described above, the region may be determined and corrected. For example, when the cross-sectional image is a CT image, the signal value in the air region approaches -1024, but the tissue such as the bronchial wall and blood vessels shows a value of 0 or more. If a low signal value region close to the air region and a high signal value close to a blood vessel etc. are included, the region of the low signal value group among the separated signal value groups is determined as the corrected cyst region. do.

Further, the signal value of the cyst region classified by the secondary classification process may be corrected by performing the threshold value process using a specific threshold value such as −950, which is the signal value of the low absorption region. ..

In addition, the area classified by the secondary classification process or the area corrected by the above-mentioned correction method is expanded by 1 unit (1 voxel or 1 pixel) toward the outside of the area, and the distribution of signal values is close. The area portion may be corrected by expanding it.

For example, in the case of the cyst region, the signal value is high in the air region in the center, but the signal value is high because the margin is a partition or fibrotic tissue of lung tissue. Therefore, it may be corrected by inflating one unit at a time until it reaches the region of high signal value. In addition, when an edge (high signal value) region is detected and the region is divided within the cyst region classified by the secondary classification process, the distribution (dispersion and distribution) of the signal value of each divided region is also obtained. By calculating and comparing the standard deviation, etc.), the region close to the lung field may be excluded and corrected.

Further, in the above description, the correction is performed using the image feature amount, but the correction is not limited to this, and the correction may be performed using the anatomical feature of the finding. Specifically, as described above, the cyst and the pleural effusion may appear in the same contour shape on the cross-sectional image, and the pleural effusion region may be misclassified as the cyst region. Therefore, if the region classified as a cyst is a peripheral region other than the lung region, that region may be deleted as a pleural effusion region. That is, the correction may be made based on the characteristics of the anatomical position of the cyst. Further, when the region classified as a cyst is a divided region r1 or r2 as shown in FIG. 13, the contour of the cyst is based on the characteristic of the anatomical shape that the contour is a continuous smooth curve. Then, the area r2 may be deleted. Also, when the region classified as a cyst is a region with protrusions as shown in FIG. 14, the protrusions are based on the anatomical shape feature that the contour of the cyst is a continuous smooth curve. You may try to delete the part of.

Further, in the above other embodiment, when the correction is performed, the cross-sectional image in the cross-sectional direction having the highest resolution is used among the plurality of cross-sectional images used when performing the primary classification process, but the present invention is not limited to this. For example, a cross-sectional image in the slice direction in photographing the subject may be used. The cross-sectional image in the slice direction at the time of shooting is a so-called original image. For example, when the slice direction at the time of shooting is a direction orthogonal to the body axis, the cross-sectional image in the axial direction is the slice direction. It becomes a cross-sectional image. Since the cross-sectional image in the slice direction is the image having the highest resolution, the classification accuracy can be further improved.

Further, the correction unit 15 performs a process of detecting an abnormality in the edge such as a protrusion or a dent in a region classified as a cyst as shown in FIG. 14, and performs the above-mentioned correction only when an abnormality in the edge is detected. You may do so. Further, when the cross-sectional images S4, S5, and S6 subjected to the secondary classification process are arranged in parallel on the three-dimensional space as shown in FIG. 15, for example, with the cyst in each cross-sectional image S4, S5, S6. When an abnormality in the continuity of the center of gravity of each of the classified regions r4, r5, and r6 is detected, the above-mentioned correction is performed for the region classified as a cyst in each of the cross-sectional images S4, S5, and S6. May be good. Regarding the abnormality of the continuity of the center of gravity position, for example, the linearity of the line L connecting the center of gravity positions is evaluated, and when the linearity is equal to or less than a preset threshold value, it is recognized that the continuity of the center of gravity position is abnormal. You just have to do it.

In this way, by correcting the secondary classification result based on the anatomical features or image features of the findings, the classification result can be made more in line with the anatomical structure, and each of the cross-sectional images can be obtained. Pixels can be classified into multiple findings with high accuracy.

In each of the above embodiments, the higher the evaluation value in all the cross-sectional images, the more the harmonic mean is adopted as a specific averaging method that preferentially classifies the findings with higher evaluation values into the findings of the common part. However, it is not limited to this.

Further, in each of the above embodiments, the primary classification result of each cross-sectional image is evaluated by different evaluation methods according to the type of findings for the pixels common to the plurality of cross-sectional images, but the evaluation is limited to the pixels. It is not something that will be done. Even if the primary classification result of each cross-sectional image is evaluated for a plurality of pixels within a predetermined range (for example, 3 × 3 × 3 pixels, 5 × 5 × 5 pixels, etc.) centered on a common pixel. good.

Further, in each of the above embodiments, the averaging method is adopted as the evaluation method, but the evaluation method is not limited to this. For example, an arbitrary evaluation method such as an evaluation method in which some parameter is changed with respect to the evaluation value for each finding included in the primary classification result can be adopted.

Further, in each of the above embodiments, the hardware of the processing unit that executes various processes such as the cross-sectional image acquisition unit 11, the primary classification unit 12, the secondary classification unit 13, the display control unit 14, and the correction unit 15. As the wear-like structure, various processors shown below can be used. As described above, the various processors include CPUs, which are general-purpose processors that execute software (programs) and function as various processing units, as well as circuits after manufacturing FPGAs (Field Programmable Gate Arrays) and the like. Dedicated electricity, which is a processor with a circuit configuration specially designed to execute specific processing such as Programmable Logic Device (PLD), which is a processor whose configuration can be changed, and ASIC (Application Specific Integrated Circuit). Circuits etc. are included.

One processing unit may be composed of one of these various processors, or a combination of two or more processors of the same type or different types (for example, a combination of a plurality of FPGAs or a combination of a CPU and an FPGA). ) May be configured. Further, a plurality of processing units may be configured by one processor.

As an example of configuring a plurality of processing units with one processor, first, as represented by a computer such as a client and a server, one processor is configured by a combination of one or more CPUs and software. There is a form in which this processor functions as a plurality of processing units. Second, as typified by System On Chip (SoC), there is a form that uses a processor that realizes the functions of the entire system including multiple processing units with one IC (Integrated Circuit) chip. be. As described above, the various processing units are configured by using one or more of the above-mentioned various processors as a hardware-like structure.

Further, as the hardware structure of these various processors, more specifically, an electric circuit (circuitry) in which circuit elements such as semiconductor elements are combined can be used.

10 Image processing device 11 Cross-sectional image acquisition unit 12 Primary classification unit 13 Secondary classification unit 14 Display control unit 15 Correction unit 20 Display device 30 Input device 40 3D image storage server 50 Multilayer neural network 51 Input layer 52 Output layer A1, A2 , A11, A12 Region B0 Boundary L Line connecting the position of the center of gravity
P pixel r1, r2, r3, r4, r5, r6 Area classified as a cyst S1, S2, S3, S4, S5, S6 Cross-sectional image

Claims (15)

A cross-sectional image acquisition unit that acquires cross-sectional images of a plurality of subjects in different cross-sectional directions,
For each of the plurality of cross-sectional images, an evaluation value indicating that each of the plurality of types of findings is obtained is obtained by performing a primary classification process for specifying the type of findings in which each pixel of each of the cross-sectional images is classified. The primary classification unit that acquires the results of the primary classification including
With respect to the portion common to the plurality of cross-sectional images, different evaluation methods are used according to the type of the findings, and the higher the evaluation value in all the cross-sectional images, the higher the evaluation value of the common portion. By evaluating the result of the primary classification process of each of the cross-sectional images by a different evaluation method including a specific averaging method that preferentially classifies the findings as one of the evaluation methods, it is common to the plurality of cross-sectional images. An image processing device provided with a secondary classification unit that performs secondary classification processing to specify the type of findings in which the parts are classified. The image processing apparatus according to claim 1 , wherein the specific averaging method is a harmonic mean. The image processing apparatus according to claim 1 or 2 , wherein the secondary classification unit performs the secondary classification process on the evaluation value of the lesion findings by the specific averaging method. The image processing apparatus according to claim 3 , wherein the secondary classification unit performs the secondary classification process on the evaluation values for findings other than the lesion by another evaluation method different from the specific averaging method. .. When the findings include a boundary between lesions and a boundary between a lesion and a tissue other than the lesion, the secondary classification unit uses the specific averaging method for the evaluation value of the findings of the boundary. The image processing apparatus according to any one of claims 1 to 4 , which performs the following classification process. The secondary classification unit according to claim 1 or 2 , wherein when the findings include a target organ, the evaluation value of the findings of the target organ is subjected to the secondary classification process by the specific averaging method. Image processing equipment. The image processing according to claim 6 , wherein the secondary classification unit performs the secondary classification process on the evaluation values of the findings other than the target organ by another evaluation method different from the specific averaging method. Device. The image processing apparatus according to claim 4 or 7 , wherein the other evaluation method is an evaluation method in which a representative value of an evaluation value in a portion common to the plurality of cross-sectional images is used as a result of the secondary classification process. From claim 1, the secondary classification unit specifies the findings obtained with the highest evaluation for the portion common to the plurality of cross-sectional images as a result of the secondary classification process as the findings regarding the common portion. 8. The image processing apparatus according to any one of 8. The image processing apparatus according to any one of claims 1 to 9 , wherein the primary classification unit performs the primary classification process using a discriminator generated by machine learning. The image processing apparatus according to any one of claims 1 to 10 , further comprising a display control unit for displaying the result of the secondary classification process on the display unit. The image processing device according to claim 11, wherein the display control unit further displays the result of the primary classification process on the display unit. The image processing apparatus according to any one of claims 1 to 12 , further comprising a correction unit for correcting the result of the secondary classification process based on the anatomical feature or the image feature amount of the finding. Obtain cross-sectional images of multiple different cross-sectional directions of the subject,
For each of a plurality of said cross-sectional images, the by row Ukoto primary classification processing for each pixel of each cross-sectional image to determine the type of findings to be classified, the evaluation value indicating that a plurality of types of findings Get the result of the primary classification including
With respect to the portion common to the plurality of cross-sectional images, different evaluation methods are used according to the type of the findings, and the higher the evaluation value in all the cross-sectional images, the higher the evaluation value of the common portion. By evaluating the result of the primary classification process of each of the cross-sectional images by a different evaluation method including a specific averaging method that preferentially classifies the findings as one of the evaluation methods, it is common to the plurality of cross-sectional images. An image processing method that performs secondary classification processing to specify the type of finding in which a part is classified. The procedure for acquiring cross-sectional images of multiple different cross-sectional directions of a subject,
For each of the plurality of cross-sectional images, an evaluation value indicating that each of the plurality of types of findings is obtained is obtained by performing a primary classification process for specifying the type of findings in which each pixel of each of the cross-sectional images is classified. The procedure for obtaining the results of the primary classification including
With respect to the portion common to the plurality of cross-sectional images, different evaluation methods are used according to the type of the findings, and the higher the evaluation value in all the cross-sectional images, the higher the evaluation value of the common portion. By evaluating the result of the primary classification process of each of the cross-sectional images by a different evaluation method including a specific averaging method that preferentially classifies the findings as one of the evaluation methods, it is common to the plurality of cross-sectional images. An image processing program that causes a computer to perform a secondary classification process that identifies the type of finding in which a part is classified.

Images