KR20230103790A - Adversarial learning-based image correction method and apparatus for deep learning analysis of heterogeneous images - Google Patents

Abstract

The present invention relates to an adversarial training-based image correction method and apparatus for deep training analysis of heterogeneous images, which comprises the steps of: generating at least one patch image from each original image based on image sets consisting of original images of different domains; generating a training dataset by selecting a portion of the at least one patch image for each original image of the image sets; identifying an RGB label and a source of the original image corresponding to each training image in the training dataset; updating the training dataset by tagging each training image with the RGB label and a domain label corresponding to the source; applying the training images of the training dataset to a feature extraction model to generate image features; and building a color constancy model by performing adversarial training of repeating attribute prediction and domain classification tasks for each training image by using the image features. Therefore, the present invention operates through the interaction of a domain classifier, which predicts the domain in which an image was taken and a lighting predictor, which predicts lighting values and removes domain characteristics by training in a way that degrades the performance of domain classification.

Description

ADVERSARIAL LEARNING-BASED IMAGE CORRECTION METHOD AND APPARATUS FOR DEEP LEARNING ANALYSIS OF HETEROGENEOUS IMAGES}

The present invention relates to an image correction technology, and more particularly, to an image color constancy model performance improvement technology based on adversarial learning of an end-to-end structure that simultaneously uses image data generated in heterogeneous environments.

Recent advances in CNN (Convolutional Neural Network) algorithms in the image field have achieved remarkable results in a wide variety of application fields, including image classification, object detection, art therapy, and movie rating prediction, both in academia and industry. However, image data, unlike text data that is created according to a consistent promise of language, is recognized as very heterogeneous in the form and structure of data depending on the shooting environment. may be expressed differently. This means that not only the different environmental information of each image, but even the image-specific information is expressed with different features, which can cause these image information to act as noise to each other and consequently act as an element that hinders the analysis performance of the model. can

In order to solve this problem, a method for improving the consistency of the data itself used for learning, that is, a method of converting an image captured in a heterogeneous environment as if it was captured in the same environment has been studied. Typically, color constancy, which removes the heterogeneity of the shooting environment by consistently expressing the color of objects that appear differently under various lights or lighting, is an essential processing process for computer vision applications. It can be widely used to create identical images.

Color constancy is largely divided into statistics-based and learning-based, and the statistical-based technique can be performed like an image preprocessing step, mainly using Intel's OpenCV library or MathWorks' MATLAB. Compared to statistics-based techniques, learning-based techniques, which have been studied relatively recently, are dominated by studies using CNNs, and can have the advantage of showing superior performance compared to existing techniques through simple networks. However, the learning-based technique also has limitations in terms of generalization, that is, when a model generated through learning of images captured in a specific environment is used for analysis of images captured in other environments, performance may be low.

Korean Patent Registration No. 10-1462440 (2014.11.11)

An embodiment of the present invention operates by interaction between a domain classifier predicting a domain, which is an environment in which an image was captured, and a lighting predictor predicting a lighting value, and removes domain characteristics through learning in a direction that deteriorates the performance of domain classification. It is intended to provide an adversarial learning-based image correction method and apparatus for deep learning analysis of heterogeneous images.

Among embodiments, an adversarial learning-based image correction method for deep learning analysis of heterogeneous images includes generating at least one patch image from each original image based on image sets each composed of original images of different domains; generating a training dataset by selecting a portion of the at least one patch image for each original image of the image sets; Identifying an RGB label and source of an original image corresponding to each training image of the training dataset; updating the training dataset by tagging each training image with the RGB label and a domain label corresponding to the source; generating image features by applying a training image of the training dataset to a feature extraction model; and constructing a color constancy model by performing adversarial learning in which an attribute prediction task and a domain classification task are repeated for each training image using the image features.

The generating of the patch image may include independently generating pixel data of each original image for each RGB channel; grouping pixels of an adjacent area defined with a specific size for the pixel data of the RGB channels; and calculating a pixel value for the RGB channel by calculating the grouped pixels.

The generating of the patch image may include determining brightness of the patch image by summing pixel values of the RGB channels.

The generating of the training dataset may include selecting top n patch images (where n is a natural number) based on brightness of the patch images.

The step of identifying the RGB label and the source may include defining an RGB value of any one of properties of the original image as the RGB label; and defining a camera characteristic associated with the original image as the source and determining the domain label according to a domain classification result of the camera characteristic.

The generating of the image feature may include applying a convolutional neural network (CNN) model that receives the training image as an input and generates the image feature as an output as the feature extraction model.

The step of generating the image feature generates a multi-dimensional feature vector of a predetermined size as the image feature through a CNN model including a convolutional layer, a max pooling layer, and a flatten layer steps may be included.

The constructing of the color constancy model may include predicting attributes of a training image by inputting the image features into an attribute prediction model including two fully connected layers; and predicting a domain of a training image by inputting the image feature into a domain classification model including one gradient reversal layer (GRL) and two fully connected layers.

Building the color constancy model may include performing the adversarial learning in a direction of increasing loss of the domain classification model while reducing loss of the attribute prediction model during the iteration.

The method may include predicting attributes of an input image using the color constancy model; and restoring an original image of the input image by removing the predicted attribute from the input image.

The method may further include evaluating performance of the color constancy model by calculating an angle between a correct answer and a predicted value for the attribute of the input image.

In one embodiment, an adversarial learning-based image correction apparatus for deep learning analysis of heterogeneous images generates at least one patch image from each original image based on image sets each composed of original images of different domains. generating unit; a learning data set generating unit generating a training data set by selecting a part from among the at least one patch image for each original image of the image sets; an original image identification unit identifying an RGB label and a source of an original image corresponding to each training image of the training dataset; a training dataset updating unit updating the training dataset by tagging each of the training images with the RGB label and a domain label corresponding to the source; an image feature generation unit generating image features by applying the learning image of the training dataset to a feature extraction model; and a model building unit configured to construct a color constancy model by performing adversarial learning by repeating attribute prediction and domain classification for each training image using the image features.

The apparatus includes an image property predictor predicting properties of an input image using the color constancy model; and an original image restoration unit configured to restore an original image of the input image by removing the predicted attribute from the input image.

The apparatus may further include a model performance evaluation unit configured to evaluate performance of the color constancy model by calculating an angle between a correct answer and a predicted value of the attribute of the input image.

The disclosed technology may have the following effects. However, it does not mean that a specific embodiment must include all of the following effects or only the following effects, so it should not be understood that the scope of rights of the disclosed technology is limited thereby.

An adversarial learning-based image correction method and apparatus for deep learning analysis of heterogeneous images according to an embodiment of the present invention operate by interaction between a domain classifier that predicts a domain in which an image is captured and a lighting predictor that predicts a lighting value. and domain characteristics can be removed through learning in the direction of degrading the performance of domain classification.

1 is a diagram illustrating an image correction system according to the present invention.
FIG. 2 is a diagram explaining the functional configuration of the image correction device of FIG. 1 .
3 is a flowchart illustrating an adversarial learning-based image correction method for deep learning analysis of heterogeneous images according to the present invention.
4 is a diagram illustrating an image correction process according to the present invention.
5 is a diagram illustrating a patch generation and label tagging process according to the present invention.
6 is a diagram illustrating a feature extraction and adversarial learning process according to the present invention.
7 is a diagram illustrating an embodiment of feature update in an adversarial learning process according to the present invention.
8 to 11 are diagrams for explaining experimental details related to the method according to the present invention.

Since the description of the present invention is only an embodiment for structural or functional description, the scope of the present invention should not be construed as being limited by the embodiments described in the text. That is, since the embodiment can be changed in various ways and can have various forms, it should be understood that the scope of the present invention includes equivalents capable of realizing the technical idea. In addition, since the object or effect presented in the present invention does not mean that a specific embodiment should include all of them or only such effects, the scope of the present invention should not be construed as being limited thereto.

Meanwhile, the meaning of terms described in this application should be understood as follows.

Terms such as "first" and "second" are used to distinguish one component from another, and the scope of rights should not be limited by these terms. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element.

It should be understood that when an element is referred to as being “connected” to another element, it may be directly connected to the other element, but other elements may exist in the middle. On the other hand, when an element is referred to as being "directly connected" to another element, it should be understood that no intervening elements exist. Meanwhile, other expressions describing the relationship between components, such as “between” and “immediately between” or “adjacent to” and “directly adjacent to” should be interpreted similarly.

Expressions in the singular number should be understood to include plural expressions unless the context clearly dictates otherwise, and terms such as “comprise” or “having” refer to an embodied feature, number, step, operation, component, part, or these. It should be understood that it is intended to indicate that a combination exists, and does not preclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

In each step, the identification code (eg, a, b, c, etc.) is used for convenience of explanation, and the identification code does not describe the order of each step, and each step clearly follows a specific order in context. Unless otherwise specified, it may occur in a different order than specified. That is, each step may occur in the same order as specified, may be performed substantially simultaneously, or may be performed in the reverse order.

The present invention can be implemented as computer readable code on a computer readable recording medium, and the computer readable recording medium includes all types of recording devices storing data that can be read by a computer system. . Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage devices. In addition, the computer-readable recording medium may be distributed to computer systems connected through a network, so that computer-readable codes may be stored and executed in a distributed manner.

All terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs, unless defined otherwise. Terms defined in commonly used dictionaries should be interpreted as consistent with meanings in the context of the related art, and cannot be interpreted as having ideal or excessively formal meanings unless explicitly defined in the present application.

Color constancy may refer to a human's ability to perceive the visual compensation that consistently recognizes the color of an object under various lighting conditions. On the other hand, since the camera expresses reflected light of an object entering through the lens as the color of the object, it can accept the color of the image expressed according to the lighting at the time of shooting. Therefore, unlike human vision, the camera may have a problem of not properly recognizing the essential characteristics of the target object due to the influence of lighting. The field of research to solve these difficulties is called color constancy or white balance, and can be largely divided into statistical-based and learning-based. In addition, the process may consist of a two-step process of estimating lighting and correcting the original image through the predicted lighting.

Traditional statistical color constancy techniques can estimate lighting by making certain statistical assumptions about image color distribution. The Gray-World algorithm can estimate lighting by calculating the average of each RGB channel, assuming that the average color of an image is achromatic. An algorithm called White-Point or Max-RGB assumes that the brightest points in the RGB channels are created by perfect reflection of light, and interprets the color of the reflection to estimate the illumination. The Gray-Edge algorithm assumes that the average of the image edges is achromatic, and calculates the average of each edge color in the RGB channels to estimate lighting. A relatively recent Grey-Pixel algorithm extracts a set of pixels whose reflectance is most similar to that of an achromatic color from an image, and can estimate the average reflectance of the corresponding pixels as lighting. Statistical-based techniques can have the advantage of being applicable with only one sheet of data due to relatively simple assumptions made through observations. However, the statistical-based method may have a limitation that it is difficult to apply when the corresponding assumption is not established, such as when color distribution is very diverse or image data without achromatic color is input. Therefore, in order to overcome these limitations, a learning-based technique that finds rules on its own in data has recently been preferred.

The learning-based technique is a method of estimating lighting through a model learned from data, and can be largely divided into Gamut Mapping and regression inference. Gamut Mapping may correspond to a method of creating a limited color space called gamut and then projecting the color space of a new input image onto the previously created space. This method works effectively when the existing color space has similar characteristics to the color space of the new image, but may have a disadvantage that performance is lowered when the characteristics are different. The regression inference method is a method of estimating the vector value of lighting by directly learning the features of an image, and machine learning models such as support vector machines and decision trees can be mainly used.

Recently, regression inference methods using deep learning have been actively attempted in an attempt to achieve maximum efficiency by utilizing the overall characteristics of learning data. Unlike previous studies that applied the CNN algorithm to color constancy, the first study that extracted statistical properties from the entire image could show excellent performance by detecting regional changes in images through patch-by-patch learning. In addition, various learning-based methods are being studied as a way to improve the performance by improving the CNN structure.

Adversarial networks, better known as generative adversarial networks, can be used to generate realistic images from meaningless noise. The adversarial network is specifically composed of generator and discriminator modules. The generator creates a fake image as similar to the real one as possible, and the discriminator minimizes maximization (which discriminates between fake and real images as accurately as possible). MiniMax) learning can be performed. Through this, the creator can create an image similar to the real one from a meaningless noise image. This dual network structure is widely used in various application fields such as image generation, fake news detection, time series synthesis, natural language processing, and domain adaptation.

Among various application fields, domain application is a subfield of transfer learning, and it can efficiently solve problems in target domains with relatively little data or no correct answer labels by utilizing a source domain with sufficient amount of data. That is, while maintaining the classification performance of source domain data, the goal is to improve classification performance in target domain data. In this process, the features of the source domain before learning and the features of the target domain appear in different distributions. Learning that overcomes this phenomenon, that is, the domain shift phenomenon, and makes the feature distribution of different domains the same It may correspond to the core of this domain application. The method of solving the domain shift is a method of inputting the target domain data into a model learned from the source domain to make the extracted features the same as the feature distribution of the source domain, and a method of simultaneously using data from two different domains for learning to achieve the two domains' A method of extracting common features among feature distributions may be included.

1 is a diagram illustrating an image correction system according to the present invention.

Referring to FIG. 1 , an image correction system 100 may include a user terminal 110 , an image correction device 130 and a database 150 .

The user terminal 110 may correspond to a computing device that is connected to the image correction device 130 to provide an image requiring image correction and to receive and check the corrected image. The user terminal 110 may be implemented as a smart phone, a laptop computer, or a computer, but is not necessarily limited thereto, and may also be implemented as various devices such as a tablet PC. The user terminal 110 may be connected to the image calibration device 130 through a network, and a plurality of user terminals 110 may be simultaneously connected to the image calibration device 130 .

The image correction device 130 may be implemented as a server corresponding to a computer or program that performs an adversarial learning-based image correction method for deep learning analysis of heterogeneous images according to the present invention. The image calibration device 130 may be connected to the user terminal 110 through a wired or wireless network and may exchange data with each other.

In one embodiment, the image correction device 130 may operate in conjunction with various external systems (or servers) in the process of performing the adversarial learning-based image correction method for deep learning analysis of heterogeneous images according to the present invention. Through this, the image correction device 130 can access various image data through SNS services, portal sites, blogs, etc., and can collect data necessary for collecting learning data for adversarial learning and building a learning model. . Also, the image correction device 130 may perform an operation of restoring and providing an original image for an input image as a response to a user's request.

The database 150 may correspond to a storage device for storing various information necessary for the operation of the image correction device 130 . For example, the database 150 may store information about a learning dataset for learning and evaluation, and may store learning algorithm and model information for building a color constancy model, but is not necessarily limited thereto, and an image calibration device. 130 may store information collected or processed in various forms in the process of performing the adversarial learning-based image correction method for deep learning analysis of heterogeneous images according to the present invention.

FIG. 2 is a diagram explaining the functional configuration of the image correction device of FIG. 1 .

Referring to FIG. 2 , the image correction device 130 includes a patch image generator 210, a training dataset generator 220, an original image identification unit 230, a training dataset updater 240, and image feature generation. It may include a unit 250, a model building unit 260, an original image restoration unit 270, a model performance evaluation unit 280, and a control unit (not shown in FIG. 2).

However, the image correction device 130 according to an embodiment of the present invention does not have to include all of the above functional components at the same time, and some of the above components may be omitted or selected from among the above components according to each embodiment. It may be implemented by selectively including some or all of them. In addition, the image correction device 130 may be implemented as independent devices that selectively include some of the above configurations as needed, and deep learning analysis of heterogeneous images according to the present invention can be performed through interworking between devices. It is also possible to perform an adversarial learning-based image correction method for Hereinafter, the operation of each component will be described in detail.

The patch image generation unit 210 may perform an operation of generating a patch image based on a data set including a plurality of images. Here, the patch image may correspond to a partial image extracted from the original original image, and a plurality of patch images may be extracted from one original image as needed. The patch image generation unit 210 may generate patch images according to a preset size and number.

Meanwhile, a data set including a plurality of images may be collected and constructed in advance, and may be composed of original images classified into different domains. That is, the data set may include a plurality of image sets, and each image set may be independently composed of images classified into a specific domain. In particular, the data set may be generated including image sets related to at least two or more different domains.

Here, the domain may represent an environment in which an image is captured. For example, a domain may be expressed as a characteristic of a camera that captures an image. That is, when image a is captured through camera A, the domain of 'image a' may be classified as 'camera A (or domain A)'. Also, when image b is captured through camera B having different characteristics from camera A, the domain of 'image b' may be classified as 'camera B (or domain B)'.

In one embodiment, the patch image generation unit 210 independently generates pixel data of each original image for each RGB channel, groups pixels in an adjacent area defined as a specific size for the pixel data of the RGB channel, and groups the pixel data of the RGB channel. A pixel value for each RGB channel may be calculated by calculating the pixels. Each pixel of the image may be expressed as a value for each of RBG, that is, red (R), green (G), and blue (B), and the patch image generator 210 generates a pixel of the RGB channel for each pixel of the original image. A value can be created, and based on this value, a patch image corresponding to a partial image of the original image can be created.

For example, if the size of the original image is 4*4, the patch image generation unit 210 may calculate pixel values of R, G, and B channels for 16 pixels and express them in a 4*4*3 array. there is. After that, the patch image generation unit 210 may group corresponding adjacent pixels by defining adjacent regions having a size of 2*2 for each of R, G, and B. In addition, the patch image generation unit 210 may generate a single patch image by combining the grouped adjacent pixels, and may determine the pixel value of the corresponding patch image by summing 4 pixel values in the adjacent area.

In one embodiment, the patch image generation unit 210 may determine the brightness of the patch image by summing pixel values for RGB channels. That is, the brightness of the patch image may be calculated by summing the pixel values of each of the RGB channels. In addition, when the patch image includes a plurality of pixels, the brightness of the corresponding patch image may be determined through the sum of pixel values of each RGB channel for each pixel.

The training dataset generation unit 220 may generate a training dataset by selecting some of a plurality of patch images extracted from each original image of the image sets. Each image set may be generated corresponding to different domains, and accordingly, patch images to be extracted from each image set may also correspond to different domains. However, the learning dataset generated by the learning dataset generation unit 220 may be configured by integrating patch images of different domains into one. The training dataset generation unit 220 may build a training dataset by using patch images as they are, but may build a training dataset by selecting only patch images that satisfy a predetermined condition among patch images.

In an embodiment, the training dataset generation unit 220 may select top n patch images (where n is a natural number) based on the brightness of the patch images. For example, when four patch images are generated from one original image, the training dataset generation unit 220 may select only the top two patch images having the brightest pixel values and add them to the training dataset. The training dataset generation unit 220 may variably determine the number of patch images selected from one original image according to characteristics of training data and learning conditions.

The original image identification unit 230 may perform an operation of identifying an RGB label and a source of an original image corresponding to each training image of the training dataset. Here, the RGB label may be expressed as a pixel value for each RGB channel of the image, and the source may correspond to the source of the image (eg, image capturing environment or camera characteristics). The original images of the image set may be created by including the RBG label and source label information in advance. That is, the original image identification unit 230 may identify the RGB label and the source by referring to the label information of the original image.

If label information is not included in the original image, the original image identification unit 230 may determine an RGB label through attribute analysis of the original image, predict camera characteristics of the original image, and classify the source. Meanwhile, the operation of the original image identification unit 230 may be performed in parallel with the operations of the patch image generator 210 and the training dataset generator 220 .

In one embodiment, the original image identification unit 230 defines any one of the RGB values of the properties of the original image as an RGB label, defines a camera characteristic associated with the original image as a source, and determines a domain classification result related to the camera characteristic. domain label can be determined accordingly. That is, an RGB label for an image may be expressed as an RGB value for a specific property of the image. For example, attributes of an image may include lighting, color, brightness, saturation, etc., and the original image identification unit 230 may specify one of the attributes and define it as an RGB label. Also, the original image identification unit 230 may classify the original image into one of predefined domains according to the characteristics of the camera in which the original image was captured. In this case, the domain may be defined based on cameras used to capture the image.

The training dataset updating unit 240 may update the training dataset by tagging each training image with an RGB label identified by the original image identification unit 230 and a domain label corresponding to the source. That is, all training images in the training dataset may be composed of image data including RGB labels and domain labels, respectively. For example, the first tag of the training image may correspond to an RGB label, and the second tag may correspond to a domain label. Meanwhile, the operation of the learning dataset updater 240 may be connected to the operation of the original image identification unit 230 and performed in parallel with the operations of the patch image generator 210 and the training dataset generator 220. It can be.

The image feature generation unit 250 may generate image features by applying the training image of the training dataset to a feature extraction model. Here, the feature extraction model may correspond to a pre-built learning model. The image feature generation unit 250 may acquire information about image features by applying each training image to an independently constructed learning model. Here, the image feature may correspond to unique feature information included in the image and may be expressed differently according to a feature extraction method applied to the training image. In addition, image features extracted from the training image may be used as training data for adversarial learning in a later step.

In one embodiment, the image feature generation unit 250 may apply a convolutional neural network (CNN) model that receives a training image as an input and generates image features as an output as a feature extraction model. In one embodiment, the image feature generator 250 generates a multidimensional feature vector of a predetermined size through a CNN model including a convolutional layer, a max pooling layer, and a flatten layer. It can be created with the above image features.

For example, the image feature generation unit 250 receives each of the patch images as an input through a CNN model, and performs a convolution operation through one convolutional layer and a max pooling layer can do. Thereafter, the CNN model can output a 3,080-dimensional high-dimensional vector through a flatten layer based on the result of the convolution operation, and the image feature generator 250 converts the high-dimensional vector output from the CNN model into an image feature can be determined by

The model builder 260 may build a color constancy model by performing adversarial learning in which the attribute prediction task and the domain classification task are repeated for each training image using image features. That is, the adversarial learning process may correspond to a process of updating model parameters related to attribute prediction and domain classification in a process in which the attribute prediction task and the domain classification task for the training image are independently performed. The adversarial learning process may be repeatedly performed until a predetermined learning goal is achieved, and at least one of models built through adversarial learning may be used as a color constancy model for removing heterogeneity of a photographing environment from an image.

In one embodiment, the model builder 260 may predict the attributes of the training image by inputting image features to an attribute prediction model including two fully connected layers. In addition, the model building unit 260 may predict the domain of the training image by inputting image features to a domain classification model including one gradient reversal layer (GRL) and two fully connected layers. The image features generated by the image feature generator 250 may be provided as inputs to an attribute prediction model and a domain classification model, respectively, and each of the attribute prediction model and the domain classification model includes an attribute prediction process and a domain classification process based on image features. can be learned through The specific process of adversarial learning is described in more detail in FIG. 6 .

The original image restoration unit 270 may restore the original image of the input image by predicting the attribute of the input image using the color constancy model and removing the predicted attribute from the input image. For example, the original image restoration unit 270 may predict the lighting properties of the input image using a color constancy model, and may restore the original image from which lighting effects are removed by removing the lighting properties from the input image. The color constancy model can effectively predict RGB values of lighting regardless of the domain of the input image, and accordingly, an original image close to the original image can be restored.

The model performance evaluation unit 280 may evaluate the performance of the color constancy model by calculating the angle between the correct answer and the predicted value of the attribute of the input image. For example, when lighting of an image is used as attribute information, the model performance evaluation unit 280 may use Angular Error (AE) as a performance evaluation index of a color constancy model. Here, each error may be expressed as an angle between actual illumination and predicted illumination, and the value of each error may decrease as the performance of the model improves.

A controller (not shown in FIG. 2 ) controls the overall operation of the image calibration device 130, and controls the patch image generator 210, the training dataset generator 220, the original image identification unit 230, and the learning data. A control flow or data flow between the set updater 240, the image feature generator 250, the model builder 260, the original image restorer 270, and the model performance evaluater 280 may be managed.

3 is a flowchart illustrating an adversarial learning-based image correction method for deep learning analysis of heterogeneous images according to the present invention, and FIG. 4 is a diagram illustrating an image correction process according to the present invention.

Referring to FIGS. 3 and 4 , in Phase 1 of FIG. 4 , the image calibration device 130 may extract images captured by the patch image generator 210 in units of patches (S310, ①), and learning data. Through the set generation unit 220, it is possible to generate a dataset for learning by selecting only necessary parts among several patches (S320, ②). At the same time, the image correction device 130 identifies the RGB label and source of each original image through the original image identification unit 230 (S330, ③), and then assigns each patch through the training dataset update unit 240. It is possible to update the learning dataset by tagging RGB labels and domain values (S340, ④).

In addition, in Phase 2 of FIG. 4 , the image calibration device 130 may perform an operation of extracting image features through CNN from the learning dataset updated through the image feature generator 250 (S350, ⑤). . Finally, the image calibration device 130 may simultaneously perform lighting prediction (A) and domain classification (B) tasks using the image features extracted in the adversarial learning step through the model building unit 260 (S360). , ⑥). Through this entire process, a color constancy model with high generalization performance can be built.

5 is a diagram illustrating a patch generation and label tagging process according to the present invention.

Referring to FIG. 5 , the upper left corner may correspond to the original image 510, and 'Domain 1', the first tag 520, may indicate the source of the image, that is, what camera captured the image. there is. In addition, 'RGB: [243, 228, 86]', which is the second tag 530, may indicate under what circumstances the corresponding image was captured, that is, the R, G, and B values of lighting used for the photographing.

In FIG. 5 , 'Image 1', which is an original image 510, is an image having a size of (4*4) and may be composed of 16 pixels. Each pixel may have a value for each of red (R), green (G), and blue (B), and information on Image 1 may be displayed in a (4*4*3) array as shown in the lower left. Next, a patch can be created by grouping adjacent pixels for each of R, G, and B, and one patch can be created by grouping 4 pixels of a size of (2*2). As a result, the information of the original image 510 can be represented in a (2*2*3) array as shown in the lower right corner, where the number displayed in each cell corresponds to the sum of the pixel values included in the cell. can

For example, 18 in the first cell of the lower right figure is a value calculated by adding 1, 2, 13, and 2 in the lower left figure. The lower right figure shows R values of 4 patches, and each patch can have G and B values as well as R values.

The next process is a process of calculating the brightness of each patch. Although omitted in FIG. 5, the total brightness of the corresponding patch can be calculated by adding the R, G, and B values of each patch. For example, if the value of the first patch in the lower right corner is R=18, G=20, and B=15, the brightness of the corresponding patch can be calculated as 53, which is the sum of pixel intensity values of RGB channels. Here, it is assumed that a patch with the brightest value is selected and used as training data in order to identify data having the greatest influence on lighting prediction. Based on this criterion, the result of selecting two patches having the largest sum of pixel intensity values of RGB channels is shown in the upper right corner.

Meanwhile, 'Patch 1' (540) and 'Patch 2' () selected in this way are both created from 'Image 1', so both of these patches inherit the label of 'Image 1' and display 'RGB: [243, 228 . '. These two patches and their respective label values can be used as input data for adversarial learning in a later process.

6 is a diagram illustrating a feature extraction and adversarial learning process according to the present invention.

Referring to FIG. 6 , an embodiment of a process of feature extraction and adversarial learning performed by the image calibration device 130 is shown. The four patches shown on the left can be generated by two each from 'Domain 1' and 'Domain 2'. The CNN 610 located in the center of FIG. 6 receives each of these patches as an input and performs a convolution operation through one convolutional layer and a max pooling layer, and then returns to the platen layer. A 3,080-dimensional high-dimensional vector can be extracted through (Flatten Layer). The feature vectors extracted in this way can be used as inputs to the domain discriminator 620 at the upper right and the illumination estimator 630 at the lower right.

The lighting predictor 630 may be configured by stacking two fully connected layers, and learning to predict a value corresponding to an RGB label may be performed. In the case of FIG. 6, 'RGB: [a, b, c]', 'RGB: [d, e, f]', 'RGB: [g, h, i]' tagged with lighting values at the time of shooting of each patch. , and 'RGB: [j, k, i]' may correspond to the target value of prediction. In one embodiment, individual images are divided into 32*32 sized patches, and only 100 patches having the highest brightness value can be used for learning. can be updated to predict

The domain classifier 620 may be composed of one 'GRL (Gradient Reversal Layer) [5]' and two fully connected layers, and each patch predicts 'Domain 1' and 'Domain 2', which are the domain labels of the patch, so that individual patches are generated. Learning to distinguish which source is generated may proceed. In this case, unlike a general deep learning model, the domain classifier 620 may be trained in a direction of increasing a loss between a correct answer and a prediction. That is, through learning, many parameters can be updated so that the 'Domain Label', which is the correct answer, cannot be distinguished well, and as a result, characteristics with high domain coherence can be removed from image data generated in a heterogeneous environment. This task can be implemented through GRL, which is a key element of adversarial learning. Specifically, during Back Propagation, the gradient received from the previous layer can be multiplied by a negative value between -1 and 0 and transmitted. there is.

Since the lighting predictor 630 and the domain classifier 620 interact simultaneously and adversarial learning is performed, the finally obtained model can well predict the RGB values of lighting regardless of the domain of the input data.

7 is a diagram illustrating an embodiment of feature update in an adversarial learning process according to the present invention.

Referring to FIG. 7, while the domain-specific characteristics of the original image are clearly distinguished in green and blue in the before feature 710, these characteristics fade as adversarial learning progresses, resulting in after-learning features (After feature). ) 730, it can be confirmed that the domain-specific characteristics have been removed. That is, the feature 730 after learning can express only the unique characteristics of an image from which the domain characteristics are excluded, and if the image is reconstructed based on this, a reconstructed image close to the actual image can be obtained. As a result, a generalizable color constancy model can be derived through adversarial learning according to the present invention.

8 to 11 are diagrams for explaining experimental details related to the method according to the present invention.

Here, experimental contents related to the image correction method according to the present invention will be described. That is, in the experiment, the performance of the color constancy model, which improves the generalization performance of the model through adversarial learning on images taken in a heterogeneous environment, can be evaluated.

Figure (a) of FIG. 9 may represent the characteristics of the benchmark dataset used in the experiment. The total number of data was 7,022, which was captured through a total of three cameras. Among them, 4,902 images taken by cameras of 'C' and 'N' companies were used for model learning, and 2,120 images taken by cameras of 'S' that were not used for learning were used for inference. If the model derived from learning the camera images of company 'C' and 'N' shows high performance in the analysis of camera images of company 'S', the model can be recognized as a model with high generalization potential.

)과 예측 RGB 값(

) 사이의 각도를 계산할 수 있으며, 모델의 성능이 우수할수록 그 값이 작게 나타날 수 있다.Angular Error, which is widely used as an evaluation index for color constancy technology, can be used to evaluate the performance of the model. The index is the RGB value of the actual lighting (Ground Truth) through the following Equation 1 (

) and predicted RGB values (

) can be calculated, and the better the performance of the model, the smaller the value may appear.

[Equation 1]

In the experiment, the performance of the comparative models (A), (B), and (C) and the model (D) according to the present invention was compared, and FIG. 8 may show the overall outline of the experiment for performance evaluation. Comparison models (A) and (B) are models in which learning is performed using only images of companies 'C' and 'N', respectively, and learning is patch-based, which is the 'feature extractor' of the method according to the present invention. It can be performed using a CNN structure. Comparison model (C) uses both 'C' and 'N' images for training, but it can simply integrate and use the two image sets without using domain information. Meanwhile, (D) is a model according to the present invention and may correspond to a model in which generalization performance is improved through adversarial learning. In order to evaluate the performance of these four models, inference can be performed on the images of company 'S', and the angular errors of the inference results of each model are compared.

The detailed structure of 'Patch-based CNN Training' in FIG. 8 follows the basic structure, but a batch normalization layer may be inserted between the convolution layer and the max pooling layer to improve generalization performance. Also, as hyperparameters, the batch size can be set to 64 and the epoch to 10. The H/W and S/W environment in which the experiment was performed is shown in Figure (b) of FIG.

In the experiment, both the comparison models and the model according to the present invention can be learned in a patch unit, and even during inference, individual inputs in a patch unit predict the RGB values of their original images, respectively. At this time, a method of calculating the angular error of the correct answer and the predicted value in each patch unit (Each Patch) or a method of calculating the angular error of the correct answer and the predicted value in each image unit by integrating the patches can be used for performance evaluation. Integration of patches can be done by aggregating the RGB values of 100 patches with the highest intensity of brightness for each image. In general, average pooling or median pooling is widely used for aggregation. can be used FIG. 10 can show the average Angular Error of each of the four models calculated according to the above three methods, and the result of this diagram is shown in FIG. 11.

10 and 11, the calculation unit of Angular Error, that is, the calculation of the patch unit and the calculation of the image unit did not show a big difference, and the Angular Error of the method according to the present invention was the lowest in all three methods. there is. This may indicate that the method according to the present invention generated a model with excellent generalization performance by removing features with high domain coherence through adversarial learning. On the other hand, the large performance difference between the comparison models (A) and (B), which were trained using only the images of 'C' and 'N' companies, respectively, can be attributed to the characteristics of the image source. In other words, the image of company 'N' used for learning was somewhat blue, and the image of company 'S' used for inference was somewhat blue-green. can be interpreted as being overestimated.

The adversarial learning-based image correction method for deep learning analysis of heterogeneous images according to the present invention can provide a method for effectively utilizing vast image data for deep learning technology in the field of computer vision. That is, although images captured in heterogeneous environments have the same information, characteristics of each image may be expressed differently depending on the capturing environment. This may mean that not only different environmental information of each image, but also image-specific information may be expressed as different features, and as a result, these image information may act as noise to hinder the analysis performance of the model. The adversarial learning-based image correction method for deep learning analysis of heterogeneous images according to the present invention provides a method for improving color constancy of images generated in a heterogeneous environment through adversarial learning, and various deep learning methods are intended to improve generalization performance. It can be widely used in running applications.

Although the above has been described with reference to preferred embodiments of the present invention, those skilled in the art will variously modify and change the present invention within the scope not departing from the spirit and scope of the present invention described in the claims below. You will understand that it can be done.

100: image correction system
110: user terminal 130: image calibration device
150: database
210: patch image generation unit 220: training dataset generation unit
230: Original image identification unit 240: Training dataset update unit
250: image feature generation unit 260: model building unit
270: original image restoration unit 280; Model performance evaluation department

Claims (14)

generating at least one patch image from each original image based on image sets each composed of original images of different domains;
generating a training dataset by selecting a portion of the at least one patch image for each original image of the image sets;
Identifying an RGB label and source of an original image corresponding to each training image of the training dataset;
updating the training dataset by tagging each training image with the RGB label and a domain label corresponding to the source;
generating image features by applying a training image of the training dataset to a feature extraction model; and
Constructing a color constancy model by performing adversarial learning by repeating the attribute prediction task and the domain classification task for each training image using the image features; deep learning analysis of heterogeneous images, including An adversarial learning-based image correction method for
The method of claim 1 , wherein generating the patch image comprises:
independently generating pixel data of each original image for each RGB channel;
grouping pixels of an adjacent area defined with a specific size for the pixel data of the RGB channels; and
An adversarial learning-based image correction method for deep learning analysis of a heterogeneous image, comprising calculating a pixel value for the RGB channel by calculating the grouped pixels.
3. The method of claim 2, wherein generating the patch image
Adversarial learning-based image correction method for deep learning analysis of heterogeneous images, comprising the step of determining the brightness of the patch image by summing the pixel values for the RGB channels.
The method of claim 3, wherein generating the training dataset
Adversarial learning-based image correction method for deep learning analysis of heterogeneous images, comprising the step of selecting the top n patch images (where n is a natural number) based on the brightness of the patch image.
2. The method of claim 1, wherein identifying the RGB label and source comprises:
defining one of RGB values among properties of the original image as the RGB label; and
Defining a camera characteristic associated with the original image as the source and determining the domain label according to a result of domain classification on the camera characteristic, adversarial learning-based image correction for deep learning analysis of heterogeneous images method.
2. The method of claim 1, wherein generating the image feature comprises:
Applying a Convolutional Neural Network (CNN) model that receives the training image as an input and generates the image features as an output as the feature extraction model. Image correction method.
7. The method of claim 6, wherein generating the image features comprises:
Generating a multi-dimensional feature vector of a preset size as the image feature through a CNN model including a convolutional layer, a max pooling layer, and a flatten layer. An adversarial learning-based image correction method for deep learning analysis of heterogeneous images.
The method of claim 1, wherein the step of building the color constancy model
predicting attributes of a training image by inputting the image features to an attribute prediction model including two fully connected layers; and
For deep learning analysis of heterogeneous images, comprising the step of predicting the domain of a learning image by inputting the image features into a domain classification model including one gradient reversal layer (GRL) and two fully connected layers Adversarial learning-based image correction method.
9. The method of claim 8, wherein the step of building the color constancy model
performing the adversarial learning in a direction of increasing a loss of the domain classification model while reducing a loss of the attribute prediction model during the iteration, and adversarial learning-based image correction for deep learning analysis of heterogeneous images method.
According to claim 1,
predicting properties of an input image using the color constancy model; and
Reconstructing the original image of the input image by removing the predicted attribute from the input image; adversarial learning-based image correction method for deep learning analysis of heterogeneous images, characterized in that it further comprises.
According to claim 11,
Evaluating the performance of the color constancy model by calculating an angle between the correct answer and the predicted value for the attribute of the input image; adversarial learning-based image correction method for deep learning analysis of heterogeneous images, characterized in that it further comprises.
a patch image generating unit generating at least one patch image from each original image based on image sets each composed of original images of different domains;
a learning data set generating unit generating a training data set by selecting a part from among the at least one patch image for each original image of the image sets;
an original image identification unit identifying an RGB label and a source of an original image corresponding to each training image of the training dataset;
a training dataset updating unit updating the training dataset by tagging each of the training images with the RGB label and a domain label corresponding to the source;
an image feature generation unit generating image features by applying the learning image of the training dataset to a feature extraction model; and
A model builder for constructing a color constancy model by performing adversarial learning by repeating attribute prediction and domain classification for each training image using the image features; deep learning of heterogeneous images including An adversarial learning-based image calibration device for analysis.
According to claim 12,
An original image restorer configured to predict a property of an input image using the color constancy model and to restore an original image of the input image by removing the predicted property from the input image; Adversarial learning-based image correction device for deep learning analysis.
According to claim 13,
A model performance evaluation unit configured to evaluate the performance of the color constancy model by calculating an angle between a correct answer and a predicted value of the attribute of the input image; adversarial learning-based image for deep learning analysis of heterogeneous images, characterized in that it further comprises correction device.

Images