KR102464851B1 - Learning method and image cassification method using multi-scale feature map - Google Patents

Abstract

A learning method and an image classification method using a multi-scale feature map extracted based on an artificial neural network are disclosed. A learning method using a multi-scale feature map includes: generating a multi-scale feature map for a training image using an artificial neural network; generating an average feature map by averaging pixel values of the multi-scale feature map in units of pixels; calculating a difference between pixel values of the average feature map and each of the multi-scale feature maps, and calculating a binary pattern code for each of the multi-scale feature maps on a pixel-by-pixel basis; and performing learning on the training image by using the average feature map and the binary pattern code.

Description

Learning method and image classification method using multi-scale feature map {LEARNING METHOD AND IMAGE CASSIFICATION METHOD USING MULTI-SCALE FEATURE MAP}

The present invention relates to a learning method and an image classification method using an artificial neural network, and more particularly, to a learning method and an image classification method using a multi-scale feature map extracted based on an artificial neural network.

Recently, the field of artificial intelligence is rapidly developing, and artificial neural networks are used to classify input images. Convolution is widely used to extract a feature map from an input image, and recently, in order to improve the performance of image classification, learning is performed by extracting a feature map of multiple scales rather than a single feature map.

There are two general methods for using multiple scales. First, the input images are converted into images of multiple scales, combined, and then applied to one deep learning model to classify the images.

Second, the input image is converted into images of multiple scales, applied to the same deep learning model, respectively, to obtain multi-scale feature maps, and then images are classified by simply combining the multi-scale feature maps. Concatenation, Addition, and Convolution are mainly used as methods for combining multi-scale feature maps.

The above-described method does not organically utilize information of multiple scales, and there is a problem in that it is difficult to properly reflect the relationship between multiple scales.

As related prior documents, there are Korean Patent Registration No. 10-2046240 and Korean Patent Publication No. 2019-0097205.

An object of the present invention is to provide a learning method and an image classification method that can more accurately classify an image using a multi-scale feature map.

Another object of the present invention is to provide a feature map extraction method capable of selecting a feature map of a useful scale for learning from a multi-scale feature map and extracting useful information.

According to an embodiment of the present invention for achieving the above object, using an artificial neural network, generating a multi-scale feature map for a training image; generating an average feature map by averaging pixel values of the multi-scale feature map in units of pixels; calculating a difference between pixel values of the average feature map and each of the multi-scale feature maps, and calculating a binary pattern code for each of the multi-scale feature maps on a pixel-by-pixel basis; and performing learning on the training image by using the average feature map and the binary pattern code.

In addition, according to another embodiment of the present invention for achieving the above object, using an artificial neural network, generating a multi-scale feature map for the input image; generating an average feature map by averaging pixel values of the multi-scale feature map in units of pixels; calculating a difference between pixel values of the average feature map and each of the multi-scale feature maps, and calculating a binary pattern code for each of the multi-scale feature maps on a pixel-by-pixel basis; and classifying the input image by using the average feature map and the binary pattern code.

In addition, according to another embodiment of the present invention for achieving the above object, using an artificial neural network, generating a multi-scale feature map for the input image; generating an average feature map by averaging pixel values of the multi-scale feature map in units of pixels; calculating a difference between pixel values of the average feature map and each of the multi-scale feature maps, and calculating a binary pattern code for each of the multi-scale feature maps on a pixel-by-pixel basis; and generating an integrated feature map to which the binary pattern code is reflected.

According to an embodiment of the present invention, a feature map of a useful scale for learning may be selected from among feature maps of several scales through binary patterning on a multi-scale feature map, and learning ability is improved by performing learning through this and image classification performance may be improved.

1 is a diagram for explaining a learning method using a multi-scale feature map according to an embodiment of the present invention.
2 is a diagram illustrating an artificial neural network according to an embodiment of the present invention.
3 is a diagram for explaining binary patterning according to an embodiment of the present invention.
4 is a diagram for explaining an image classification method using a multi-scale feature map according to an embodiment of the present invention.
5 is a diagram for explaining an image classification result according to an embodiment of the present invention.
6 is a diagram for explaining a method for extracting a feature map for image classification according to an embodiment of the present invention.

Since the present invention can have various changes and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention. In describing each figure, like reference numerals have been used for like elements.

The present invention relates to a learning method for image classification and a method for classifying an input image based on machine learning, and uses a multi-scale feature map to increase classification accuracy for an input image.

An embodiment of the present invention organically combines the feature values of the input image included in the multi-scale feature map, extracts useful information from the feature values included in the multi-scale feature map, and uses it for input image classification, Binary pattern the multi-scale feature map.

The learning method and the image classification method using a multi-scale feature map according to an embodiment of the present invention can be utilized in various fields for classifying an input image into a preset class, and includes a processor and a memory such as a desktop, a laptop computer, a server, etc. may be performed on a computing device.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram for explaining a learning method using a multi-scale feature map according to an embodiment of the present invention, and FIG. 2 is a diagram illustrating an artificial neural network according to an embodiment of the present invention.

1 and 2 , a computing device performing a learning method according to an embodiment of the present invention generates a multi-scale feature map 220 for a training image 210 for learning by using an artificial neural network. (S110). That is, the computing device generates the feature maps 220 of different scales, and as an embodiment, the feature map 220 may be extracted from the training image 210 in a convolution layer (L1).

The computing device may generate a multi-scale feature map by using kernels of different sizes or by adjusting the number of convolutions for the training image. Here, the scale may be a concept including both the size and resolution of the feature map, and the multi-scale feature map may mean a plurality of feature maps having different sizes and resolutions. In FIG. 2 , an embodiment in which five multi-scale feature maps 220 having different sizes and resolutions are used is described.

In addition, the computing device may convert the size of the multi-scale feature map to be the same as the size of one of the multi-scale feature maps 220 in step S110. As the size of the feature map becomes the same, the resolution of the feature map becomes the same. can

As an embodiment, the computing device may adjust the size of the feature map by using a transition layer (L2), and the transition layer may include a 1x1 convolutional layer and a resizing layer. Through the 1x1 convolutional layer, the number of channels of the multi-scale feature map can be processed to be the same, and the sizes of the first, second, fourth, and fifth feature maps 221, 222, 224, 225 are reduced through the resizing layer. 3 It may be processed to be the same as the size of the feature map 223 . The resizing layer may convert the size of the feature map by averaging neighboring pixel values or interpolating neighboring pixel values.

And, the computing device performs learning on the training image by using the multi-scale feature map. As an embodiment, the computing device may perform learning by binary patterning the multi-scale feature map using the binary patterning layer L3.

More specifically, the computing device generates an average feature map by averaging the pixel values of the multi-scale feature map in units of pixels ( S120 ). Then, the difference between the pixel values of the average feature map and the multi-scale feature map is calculated, and a binary pattern code for each of the multi-scale feature maps is calculated in units of pixels (S130). Steps S130 and S140 may be performed in the SBP encoder, and steps S130 and S140 are described in more detail in FIG. 3 .

The computing device performs learning on the training image by using the average feature map and the binary pattern code (S140). By giving a class label to the training image, the artificial neural network can learn what kind of image the training image is.

As an embodiment, the computing device may learn at least one of the presence or absence of a tumor and the degree of differentiation of the tumor with respect to the training image. In the case of detecting a tumor, an image containing tumor cells and an image consisting only of normal cells may be used as training images. can be used And the presence or absence of tumor cells or the degree of differentiation of the tumor may be given as a class label.

In step S140, the computing device convolves and max-pools each of the average feature map 240 and the integrated feature map 230 obtained from the binary pattern code, and then adds the final feature map 250 obtained by combining to the training image 210. It can be used as a feature map for learning. That is, the final feature map is input to a fully connected neural network, so that learning can be performed.

3 is a diagram for explaining binary patterning according to an embodiment of the present invention.

As shown in FIG. 3 , in the computing device performing the learning method according to an embodiment of the present invention, in step S120, the k-th pixel at the same location of the multi-scale feature maps 321 to 325 converted to the same size. The values 326 are averaged, and the average feature map 240 having the average value 327 as the k-th pixel value is generated. Since the size of the average feature map 240 is the same as the size of the transformed multi-scale feature maps 321 to 325, a plurality of pixel values of the multi-scale feature maps 321 to 325 are averaged to generate one pixel value. , one average feature map 240 is generated from the multi-scale feature maps 321 to 325 .

)에 대한 평균값(

)은 [수학식 1]과 같이 계산될 수 있다.The k-th pixel value of the multi-scale feature maps 321 to 325 (

) for the average (

) can be calculated as in [Equation 1].

)은 평균 특징맵의 k번째 픽셀값에 대응된다.Here, j is the index of the multi-scale feature map, S represents the number of multi-scale feature maps, and c represents the channel of the multi-scale feature map. And the average value of the kth pixel value (

) corresponds to the k-th pixel value of the average feature map.

Then, the computing device calculates the difference between the k-th pixel value 326 of the multi-scale feature map 321 to 325 and the k-th pixel value 327 of the average feature map 240 in step S130, and the calculated difference value Therefore, it is possible to calculate a binary pattern code for each of the multi-scale feature maps in units of pixels. As an embodiment, the binary pattern code may be assigned a value of 1 when the pixel value difference is equal to or greater than 0, and 0 when the difference between the pixel values is less than 0.

3, the difference between the k-th pixel value of the first and second feature maps 321 and 322 and the k-th pixel value of the average feature map 240 is less than 0, and the third to fifth feature maps 323, The difference between the k-th pixel value of 324 and 325 and the k-th pixel value of the average feature map 240 is 0 or more. Therefore, the binary pattern code for the k-th pixel becomes 00111.

In this case, each digit of the binary number constituting the binary pattern code may be determined according to each scale of the multi-scale feature map. As an example, as shown in FIG. 3 , as the size of the scale increases, the position of the corresponding binary number may be determined in the order of the highest digit to the lowest digit.

Since the scale of the first feature map 321 is the smallest, the binary 0 obtained from the first feature map 321 corresponds to the most significant digit of the binary pattern code, and since the scale of the fifth feature map 325 is the largest, The binary number 1 obtained from the fifth feature map 325 corresponds to the lowest digit of the binary pattern code.

In this way, when the binary pattern code is calculated in units of pixels of the multi-scale feature map, the computing device converts the binary pattern code into a single decimal number in units of pixels to generate the integrated feature map 230 , and the average feature map 240 ) and the integrated feature map 230 to perform learning on the training image. That is, the binary pattern code for the k-th pixel is converted into one decimal number, and the k-th pixel value of the integrated feature map 230 corresponds to the converted decimal number. The average feature map 240 and the integrated feature map 230 have the same size.

As an embodiment, the computing device may calculate a binary pattern code and convert the binary pattern code into a decimal number by using [Equation 2].

는 다중 스케일 특징맵의 c채널에 대한 통합 특징맵(230)의 k번째 픽셀의 픽셀값을 나타내며,

는 다중 스케일 특징맵의 c채널에 대한 평균 특징맵(240)의 k번째 픽셀의 픽셀값을 나타낸다. j는 다중 스케일 특징맵의 인덱스로서, 5개의 다중 스케일 특징맵(321 내지 335)에 대해 1에서 5사이의 자연수 중에서, 스케일이 작을수록 큰 값이 할당될 수 있다.here,

represents the pixel value of the k-th pixel of the integrated feature map 230 for the c channel of the multi-scale feature map,

denotes the pixel value of the k-th pixel of the average feature map 240 for the c channel of the multi-scale feature map. j is an index of the multi-scale feature map, among natural numbers between 1 and 5 for the five multi-scale feature maps 321 to 335, a larger value may be assigned as the scale is smaller.

In FIG. 3 , since the binary pattern code for the k-th pixel is 00111, the pixel value of the k-th pixel of the integrated feature map 230 is 7 in decimal.

Then, the integrated feature map 230 is combined with the average feature map 240, and then is input to a fully connected artificial neural network and used for learning.

As described above, according to an embodiment of the present invention, 1 of the binary pattern codes is reflected in a decimal number, which means that among the multi-scale feature maps, a feature map of a scale corresponding to 1 is used for learning as useful information. That is, in the above-described embodiment, it can be seen that the third to fifth feature maps 323 , 324 , and 325 obtained by obtaining the binary number 1 are useful information and are used for learning.

After all, according to an embodiment of the present invention, a feature map of a useful scale for learning can be selected from among feature maps of multiple scales through binary patterning on a multi-scale feature map, and learning ability is performed by performing learning through this. This can be improved.

4 is a diagram for explaining an image classification method using a multi-scale feature map according to an embodiment of the present invention.

An image classification method according to an embodiment of the present invention classifies an input image into one of preset classes using the artificial neural network learned in FIGS. 1 to 3 . Therefore, similar to the above-described learning process, a multi-scale feature map is generated and binary patterning is performed.

Referring to FIG. 4 , a computing device performing an image classification method according to an embodiment of the present invention generates a multi-scale feature map for an input image using an artificial neural network ( S410 ). The computing device may convert the size of the multi-scale feature map to be the same as the size of one of the multi-scale feature maps.

Then, the computing device generates an average feature map by averaging the pixel values of the multi-scale feature map in units of pixels (S120).

Then, the computing device calculates the difference between the pixel values of the average feature map and the multi-scale feature map, and calculates a binary pattern code for each of the multi-scale feature maps in units of pixels (S130). At this time, each digit of the binary number constituting the binary pattern code may be determined according to the scale of each multi-scale feature map, and the binary pattern code is a code in which 1 is assigned when the difference between pixel values is 0 or more, and 0 is assigned when it is less than 0. can be

In addition, the computing device may classify the input image by using the average feature map and the binary pattern code ( S140 ), and classify the input image into one of the class labels used in the learning process. As an embodiment, the computing device may classify the input image into a class according to the presence or absence of a tumor or classify the input image into a class according to the differentiation degree of the tumor.

The computing device may convert the binary pattern code into a single decimal number in units of pixels, generate an integrated feature map, combine the average feature map and the integrated feature map, and classify the input image.

5 is a diagram for explaining an image classification result according to an embodiment of the present invention.

The learning method and the image classification method according to an embodiment of the present invention can be used to classify images generated in various fields such as security and transportation, and can also be used in the medical field for detecting tumors and predicting the degree of differentiation of tumors. can

In the case of detecting a tumor, an image containing tumor cells and an image consisting only of normal cells may be used as training images. When predicting the grade of tumor, images corresponding to different degrees of differentiation are used as training images. can be The degree of differentiation refers to a degree of similarity between tumor cells and normal cells, and a high degree of differentiation indicates a high degree of similarity between tumor cells and normal cells.

FIG. 5 is a view showing a detection result and a differentiation degree prediction result for colorectal cancer, and shows the results of learning and testing with 11294 image patches of 1024x1024 size generated from colorectal cancer tissue images. Among the 11294 image patches, the training data set consists of 1525 normal cell image patches, 1684 low-differentiated tumor image patches, 2925 medium-differentiated tumor image patches, and 1358 highly differentiated tumor image patches. is composed of 1512 normal cell imaging patches, 638 low-differentiation tumor imaging patches, 1180 medium-differentiation tumor imaging patches, and 472 high-differentiation tumor imaging patches.

5 shows the results of using the artificial neural network ResNet50 and the artificial neural network MobileNetV1. In Figure 5, ResNet detects and predicts results using a single-scale feature map, FF _concat -ResNet combines multi-scale maps by concatenation method to detect and predict the results, FF _add -ResNet adds a multi-scale map As a result of detection and prediction by combining methods, FF _conv -ResNet shows detection and prediction results by combining multi-scale maps with a convolution method, and MSBP-Net-ResNet shows detection and prediction results according to an embodiment of the present invention. .

And in Figure 5, MobileNet detects and predicts using a single-scale feature map, FF _concat -MobileNet combines multi-scale maps by concatenation method to detect and predict, FF _add -MobileNet adds a multi-scale map As a result of detection and prediction by combining methods, FF _conv -MobileNet shows detection and prediction results by combining multi-scale maps with a convolution method, and MSBP-Net-ResNet shows detection and prediction results according to an embodiment of the present invention. .

As shown in FIG. 5 , it can be seen that the tumor detection accuracy (ACC _BvsC ) and differentiation degree prediction accuracy (ACC _Grade ) according to an embodiment of the present invention exhibit the highest accuracy at 99.45% and 86.61%. Tumor detection (F ₁ ^BN ) and low differentiation (F ₁ ^WD ), medium differentiation (F ₁ ^MD ), and high differentiation (F ₁ ^PD ) according to an embodiment of the present invention F1 scores of the prediction results are 0.9930, 0.7399, 0.7881 , 0.8337, indicating that the tumor detection and differentiation degree prediction results according to an embodiment of the present invention are the best.

6 is a diagram for explaining a method for extracting a feature map for image classification according to an embodiment of the present invention.

A computing device for performing the feature map extraction method according to an embodiment of the present invention generates a multi-scale feature map for an input image using an artificial neural network (S610), and calculates pixel values of the multi-scale feature map in units of pixels. By averaging, an average feature map is generated (S620). Then, the difference between the pixel values of the average feature map and the multi-scale feature map is calculated, and a binary pattern code for each of the multi-scale feature maps is calculated in units of pixels (S630). Then, an integrated feature map S640 in which the calculated binary pattern code is reflected is generated.

In step S640, the computing device converts the binary pattern code into a single decimal number in units of pixels to generate an integrated feature map, and each digit of the binary number constituting the binary pattern code is determined according to the scale of each multi-scale feature map. In addition, the binary pattern code may be a code in which 1 is assigned when the pixel value difference is greater than or equal to 0, and 0 is assigned when the difference between pixel values is less than 0.

The generated integrated feature map may be used for image classification or learning for image classification, and as an embodiment, as described above, may be used in combination with the average feature map.

The technical contents described above may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiments, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. A hardware device may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, the present invention has been described with specific matters such as specific components and limited embodiments and drawings, but these are only provided to help a more general understanding of the present invention, and the present invention is not limited to the above embodiments. , various modifications and variations are possible from these descriptions by those of ordinary skill in the art to which the present invention pertains. Therefore, the spirit of the present invention should not be limited to the described embodiments, and not only the claims to be described later, but also all those with equivalent or equivalent modifications to the claims will be said to belong to the scope of the spirit of the present invention. .

Claims (15)

In a learning method using a multi-scale feature map, performed on a computing device,
generating a multi-scale feature map for a training image using an artificial neural network, and transforming the size of the multi-scale feature map to be the same as the size of one of the multi-scale feature maps;
generating an average feature map by averaging pixel values of the converted multi-scale feature map in units of pixels;
calculating the difference between the pixel values of the average feature map and the transformed multi-scale feature map, and calculating a binary pattern code for each of the transformed multi-scale feature maps in units of the pixel according to the calculated difference value step; and
using the average feature map and the binary pattern code to perform learning on the training image,
The step of performing learning on the training image is
generating an integrated feature map by converting the binary pattern code into a single decimal number in the pixel unit; and
performing the learning by combining the average feature map and the integrated feature map,
Each digit of the binary number constituting the binary pattern code is
determined according to each scale of the multi-scale feature map
A learning method using multi-scale feature maps.
delete delete The method of claim 1,
The binary pattern code is
A code in which 1 is assigned when the difference between the pixel values is greater than or equal to 0, and 0 is assigned when the difference between the pixel values is less than 0.
A learning method using multi-scale feature maps.
delete The method of claim 1,
The step of performing learning on the training image is
Learning at least one of the presence or absence of a tumor for the training image and the degree of differentiation of the tumor
A learning method using multi-scale feature maps.
In the image classification method using a multi-scale feature map, performed in a computing device,
generating a multi-scale feature map for an input image using an artificial neural network, and transforming the size of the multi-scale feature map to be the same as the size of one of the multi-scale feature maps;
generating an average feature map by averaging pixel values of the converted multi-scale feature map in units of pixels;
calculating the difference between the pixel values of the average feature map and the transformed multi-scale feature map, and calculating a binary pattern code for each of the transformed multi-scale feature maps in units of the pixel according to the calculated difference value step; and
classifying the input image by using the average feature map and the binary pattern code,
The step of classifying the input image is
generating an integrated feature map by converting the binary pattern code into a single decimal number in the pixel unit; and
classifying the input image by combining the average feature map and the integrated feature map,
Each digit of the binary number constituting the binary pattern code is
determined according to each scale of the multi-scale feature map
An image classification method using a multi-scale feature map.
delete delete 8. The method of claim 7,
The binary pattern code is
A code in which 1 is assigned when the difference between the pixel values is greater than or equal to 0, and 0 is assigned when the difference between the pixel values is less than 0.
An image classification method using a multi-scale feature map.
delete 8. The method of claim 7,
The step of classifying the input image is
Classifying the input image according to at least one of the presence or absence of a tumor and the degree of differentiation of the tumor
An image classification method using a multi-scale feature map.
In the feature map extraction method for image classification, performed in a computing device,
generating a multi-scale feature map for an input image using an artificial neural network, and transforming the size of the multi-scale feature map to be the same as the size of one of the multi-scale feature maps;
generating an average feature map by averaging pixel values of the converted multi-scale feature map in units of pixels;
calculating a difference between the pixel values of the average feature map and the transformed multi-scale feature map, and calculating a binary pattern code for each of the multi-scale feature maps on a pixel-by-pixel basis according to the calculated difference value; and
and generating an integrated feature map in which the binary pattern code is reflected,
The step of generating the integrated feature map is
converting the binary pattern code into a single decimal number in the pixel unit to generate the integrated feature map,
Each digit of the binary number constituting the binary pattern code is
determined according to each scale of the multi-scale feature map
A feature map extraction method for image classification.
delete 14. The method of claim 13,
The binary pattern code is
A code in which 1 is assigned when the difference between the pixel values is greater than or equal to 0, and 0 is assigned when the difference between the pixel values is less than 0.
A feature map extraction method for image classification.

Images