KR102475177B1 - Method and apparatus for processing image - Google Patents

Abstract

The present invention relates to an image processing method and apparatus. An image processing method according to an embodiment of the present invention includes query image information (q) and N pieces of search target image information (s _{1 to s N ) (hereinafter, s 1 to} s , where _N is a natural number of 2 or more) a generating step of performing nonlinear data dimensionality reduction on each of _N (referred to as s _i ) to generate q' and s _i ' representing at least one element with respect to an attribute of a corresponding image; and a comparison step of indirectly performing a comparison with q for each s _i using dimensionally reduced q' and s i _' .

Description

Image processing method and apparatus {METHOD AND APPARATUS FOR PROCESSING IMAGE}

The present invention relates to an image processing method and apparatus, and more particularly, to an image processing method and apparatus capable of performing mutual comparison between given image information through a preprocessing technique such as supporting semantic variable control.

Among various image processing, there is a case where a technique for comparing mutual similarity between a plurality of images (hereinafter, referred to as “image comparison processing”) is required. That is, in the image comparison process, when image information (q) corresponding to a query is input, a set (S) corresponding to a search space or gallery (however, S={s ₁ , s ₂ , ...s _i , ...s _N }, i∈{1, 2, ...N}, where N is a natural number greater than or equal to 2), is a technique for comparing mutual similarity between each element and the query image information (q). Hereinafter, S is referred to as a “search target set”, and elements s ₁ to s _N of the search target set S are referred to as “search target image information”.

Such image comparison processing frequently occurs in data search, comparison, and learning. For example, when query image information (q) is given, “similarity (q, s _{i )} ” (provided that i∈{1, 2, …N}). At this time, if similarity (q, s _i ) can be calculated, the similarity is calculated for each of the N elements of the set to be searched (S), and then a certain condition (eg, the highest or lowest similarity) A process of finding a subset of the search target set (S) that satisfies a condition such as, being included in a critical range, or not included in a critical range may be required in various applications.

<Problem when query image information (q) is corrupted>

On the other hand, in reality, a case may occur in which an element of the query image information (q) or the search target set (S) is obscured by an arbitrary object, noise is generated, damaged, or too small. In this way, cases in which the image is contaminated or the information is insufficient due to a specific cause frequently occur. Conventionally, in this case, it is difficult to completely compare the query image information (q) and the elements of the set to be searched (S).

<The problem of difficult comparison of semantic level>

In addition, conventionally, q and S = {s ₁ , s ₂ , . . . s _i , . . . s _N }, comparison metrics such as mean squared error (MSE) and peak to signal to noise ratio (PSNR) were used. However, although these metrics are effective at the level of judging the degree to which pixel values match each other, they are not designed to reflect and compare higher-level object characteristics. In order to improve this problem, a metric such as SSIM (Structural Similarity Index Measure) (hereinafter referred to as “improvement metric”) is designed to reflect the structural similarity existing in an image. However, even in the case of this improvement metric, it does not perform comparison at the level of semantics (meaning) that people usually think.

Here, “comparison at the semantic level” means a comparative analysis between meanings at a much higher level than simple comparison between pixels, such as ‘relationship between pink clothes and red clothes’ and ‘relationships between pink clothes and pink flowers’. do. Perhaps, such a semantic level comparison may be performed using various conventional techniques. However, such comparison processing usually increases the amount of computation, resulting in problems such as slow processing speed, high energy consumption, and inefficient use of computational resources.

KR 10-2013-0108425 A

In order to solve the problems of the prior art as described above, the present invention provides an image comparison processing technique between a query input image and a search target set, but by actively and selectively changing high-level feature elements present in the image, semantic level Its purpose is to provide a new type of image comparison processing technology capable of variable control.

Another object of the present invention is to provide an image comparison processing technology capable of efficiently performing calculation processing.

However, the problem to be solved by the present invention is not limited to the above-mentioned problems, and other problems not mentioned can be clearly understood by those skilled in the art from the description below. There will be.

In order to solve the above problem, an image processing method according to an embodiment of the present invention includes query image information (q) and N (N is a natural number of 2 or more) search target image information (s ₁ to s _N ) (hereinafter, s ₁ to s _N are referred to as s _i ) by performing non-linear data dimensionality reduction, respectively, to generate q 'and s _i 'represented by at least one element for the attribute of the image step; and a comparison step of indirectly performing a comparison with q for each s _i using dimensionally reduced q' and s i _' .

The q' and _si ' may be generated using a pre-learned model using a machine learning technique.

The model may each include an input layer through which input data is input, an output layer through which output data is output, and at least one hidden layer connecting the input layer and the output layer with a reduced number of nodes compared to the input layer and the output layer.

The q' and s _i ' may be generated based on at least one hidden layer in which node values are set by inputting q and s _i to the model, respectively.

In the comparing step, q' and s _i ' may be directly compared, or elements of at least one attribute included in q' and s _i ' may be compared.

The comparing step may include comparing at least one of q' and s _i ' after additionally processing it within a corresponding reduced dimension.

The additional processing may include changing an element value of q' or s _i ' related to at least one attribute.

The image processing method according to an embodiment of the present invention may further include an extraction step of extracting information about attributes of q and s _i .

The comparison step may include a step of additionally processing at least one of q' and s _i ' within the corresponding reduced dimension using the information extracted in the extraction step.

The additional processing may include removing at least one contaminated attribute from q from q′, or removing at least one contaminated attribute from one s _i from the corresponding s _i .

The further processing may include injecting at least one of the contaminated attributes in q into each s _i '.

The comparison step may include controlling variables by selectively changing attributes of q' or s _i '.

The comparing step may include decoding and comparing q' and s _i ', or decoding and comparing q' or s _i ' after additional processing within a corresponding reduced dimension.

In the generating step, at least one attribute contaminated in q is removed from q' or at least one attribute contaminated in any one s _i is removed from q' through a pre-learned model using a machine learning technique. performing the data dimensionality reduction while removing from _i '.

The generating step may include generating a plurality of q' representing different attributes, and generating a plurality of each of s ₁ 'to s _N ' to correspond to attributes according to each q'.

An image processing apparatus according to an embodiment of the present invention includes query image information (q) and N (where N is a natural number of 2 or more) search target image information (s ₁ to s _N ) (hereinafter, s ₁ to s a memory storing _N (referred to as s _i ); and a control unit controlling a comparison of each s _i with q using q and s _i stored in the memory.

In addition, the image processing apparatus according to an embodiment of the present invention includes query image information (q) and N (where N is a natural number of 2 or more) search target image information (s ₁ to s _N ) (hereinafter, s ₁ to s _N is referred to as s _i ) from the first device; and a control unit controlling a comparison with q for each s _i using the q and s _i received by the communication unit, and controlling transmission of the comparison result to the first device or the second device.

The control unit performs non-linear data dimensionality reduction on q and _si , respectively, to generate q' and s _i ' representing at least one element with respect to the attribute of the corresponding image, and dimensionally reduced q' and s _i Comparison with q for each s _i can be performed indirectly using '.

An image processing apparatus according to another embodiment of the present invention includes query image information (q) and N (N is a natural number of 2 or more) search target image information (s ₁ to s _N ) (hereinafter, s 1 to s ₁ to s N ). a memory storing s _N as s _i ); a non-linear dimensional conversion unit for performing non-linear data dimensionality reduction on q and s _i , respectively, to generate q' and s _i ' representing at least one element with respect to an attribute of a corresponding image; and a comparison unit that performs a comparison with q for each s _i using q' and s _i '.

An image processing apparatus according to another embodiment of the present invention includes query image information (q) and N (where N is a natural number of 2 or more) search target image information (s ₁ to s _N ) (hereinafter, s ₁ to s _N (referred to as s _i ) from the first device and transmits a comparison result of the comparison unit to the first device or the second device; a non-linear dimensional conversion unit for performing non-linear data dimensionality reduction on q and s _i , respectively, to generate q' and s _i ' representing at least one element with respect to an attribute of a corresponding image; and a comparison unit that performs a comparison with q for each s _i using q' and s _i '.

The present invention configured as described above provides an image comparison processing technology between a query input image and a search target set, but a new type of image capable of semantic variable control by actively and selectively changing high-level feature elements present in the image. There is an advantage that comparison processing is possible.

In addition, the image comparison processing according to the present invention efficiently performs calculation processing through data dimension reduction, enabling faster comparison processing compared to comparison processing in the image domain having a large size and reducing the amount of stored data. There is an advantage.

In addition, the present invention has an advantage that it can be used in various fields that require image search, similarity comparison, classification, and the like.

The effects obtainable in the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below. will be.

1 shows an overall conceptual diagram of image comparison processing according to the present invention.
2 shows a block diagram of an image processing device 100 according to an embodiment of the present invention.
3 shows a block diagram of the controller 150 in the image processing device 100 according to an embodiment of the present invention.
4 is a flowchart of an image processing method according to an embodiment of the present invention.
5 shows a conceptual diagram for image comparison using nonlinear dimensionality reduction processing according to the present invention.
6 shows an example of face comparison according to the present invention.
7 shows another example of face comparison according to the present invention.
8 shows an example of comparison after q and s _i are each expressed with a plurality of expression vectors in the present invention.

The above objects and means of the present invention and the effects thereof will become clearer through the following detailed description in conjunction with the accompanying drawings, and accordingly, those skilled in the art to which the present invention belongs can easily understand the technical idea of the present invention. will be able to carry out. In addition, in describing the present invention, if it is determined that a detailed description of a known technology related to the present invention may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted.

Terms used in this specification are for describing the embodiments and are not intended to limit the present invention. In this specification, singular forms also include plural forms in some cases unless otherwise specified in the text. In this specification, terms such as "comprise", "have", "provide" or "have" do not exclude the presence or addition of one or more other elements other than the mentioned elements.

In this specification, terms such as “or” and “at least one” may represent one of the words listed together, or a combination of two or more. For example, "or B" and "at least one of B" may include only one of A or B, or may include both A and B.

In this specification, descriptions following "for example" may not exactly match the information presented, such as cited characteristics, variables, or values, and tolerances, measurement errors, limits of measurement accuracy and other commonly known factors It should not be limited to the embodiments of the present invention according to various embodiments of the present invention with effects such as modifications including.

In this specification, when a component is described as being 'connected' or 'connected' to another component, it may be directly connected or connected to the other component, but there may be other components in the middle. It should be understood that it may be On the other hand, when a component is referred to as 'directly connected' or 'directly connected' to another component, it should be understood that no other component exists in the middle.

In the present specification, when an element is described as being 'on' or 'in contact with' another element, it may be in direct contact with or connected to the other element, but another element may be present in the middle. It should be understood that On the other hand, if an element is described as being 'directly on' or 'directly in contact with' another element, it may be understood that another element in the middle does not exist. Other expressions describing the relationship between components, such as 'between' and 'directly between', can be interpreted similarly.

In this specification, terms such as 'first' and 'second' may be used to describe various elements, but the elements should not be limited by the above terms. In addition, the above terms should not be interpreted as limiting the order of each component, and may be used for the purpose of distinguishing one component from another. For example, a 'first element' may be named a 'second element', and similarly, a 'second element' may also be named a 'first element'.

Unless otherwise defined, all terms used in this specification may be used in a meaning commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless explicitly specifically defined.

Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the accompanying drawings.

1 shows an overall conceptual diagram of image comparison processing according to the present invention.

The present invention relates to a technique for processing image information for comparison between images. Referring to FIG. 1, query image information (q) and a search target set (S) (however, S={s ₁ , s ₂ , …s _i , …s _N }, i∈{1, 2, …N} , N is a natural number of 2 or more), q' and S' (however, S'={s ₁ ^' , s ₂ ^' , …s _i ^' , …s _N ^' } through the image information processing of the present invention) , i ∈ {1, 2, ... N}) can be output. In this case, the query image information (q) is image information requested by the query, and the search target set (S) is a set of search target image information (s ₁ to s _N ) to be compared with the query image information (q).

In addition, the query image information (q) and q' may have the same or different data dimensions, and each image information (s ₁ to s _N ) existing in the search target set (S) is S' Dimensions of each element (s ₁ 'to s _N ') and its data may be the same or different. For example, if the input image is 100x100, the output image may be 100x100, 1000x1000, or 10x10. In summary, through image information processing, the size of an output image with respect to an input image may be the same, may be increased, or may be reduced.

In particular, it is an essential part of the present invention to perform image information processing in which the dimension of output data is reduced compared to the size of the input data (hereinafter referred to as “dimensional reduction processing”). This is because the efficiency of calculation processing can be increased and the required storage space can be reduced through dimensionality reduction processing.

That is, through the dimensionality reduction process, q' may have information whose dimension of the data is reduced compared to q, and the element of S' (ie, s _i ') is the data compared to the element of S (s _i ). The dimension of may have reduced information.

For example, through dimensionality reduction processing, image information A may be output as A′ in which the dimensionality of data is reduced. At this time, the information of A may be expressed as a vector having a number of elements, a 2D matrix, or a 3D matrix. In addition, the information of A' may be expressed as a vector having a' number of elements (where a>a'), a 2D matrix, or a 3D matrix. In this case, through dimensionality reduction processing, A of the 3D matrix is processed into a 3D matrix, 2D matrix or vector A' of a smaller scale, so that the data dimension can be reduced. In addition, through dimensionality reduction processing, A of the 2D matrix is processed into A', which is a 2D matrix or vector of a smaller scale, so that the data dimension can be reduced.

However, in the present invention, after the dimension reduction process, image information is processed (hereinafter referred to as " Performing dimension expansion processing”) may be optionally added.

Meanwhile, in the present invention, query image information (q) may be compared with elements (s ₁ to s _N ) of the search target set (S), respectively. However, the present invention does not directly compare q for each s _i , but generates q' and s _i ' through dimension reduction processing for q and s _i , and then calculates the dimensionally reduced q' and s Comparison with q for each s _i is performed indirectly using _i '. That is, in the present invention, indirect comparison with q for each s _i can be performed by comparing q' and s _i ', or comparing one of q' and s _i ' with additional processing (image information processing). .

In FIG. 1, q' and S' output through image information processing are modified to facilitate comparison of semantic levels with each other. However, q is indicated as one for clarity, but may be more than one.

Calculation of the mutual similarity between the query image information (q) and the elements (s ₁ to s _N ) of the search target set (S) has conventionally been a necessary process in many applications. In relation to this process, in the present invention, by performing indirect comparison through image information processing, <removal of contaminating elements of query image information (q)>, <removal of contaminating elements present in the search target set (S)>, and <search Injection of contamination elements present in query image information (q) into each image information (s ₁ to s _N ) of the target set (S)>, <Semantic level between query image information (q) and search target set (S) , <support for comparison of query image information (q) and search target set (S)>, <support for active variable control at semantic level>. A detailed description of these will be described later.

2 shows a block diagram of an image processing device 100 according to an embodiment of the present invention.

The image processing device 100 according to an embodiment of the present invention is a device that processes image information for comparison between images, and may be an electronic device capable of computing or a computing network.

For example, the electronic device includes a desktop personal computer, a laptop personal computer, a tablet personal computer, a netbook computer, a workstation, and a personal digital assistant (PDA). , a smart phone (smartphone), a smart pad (smartpad), or a mobile phone (mobile phone), etc., but is not limited thereto.

As shown in FIG. 2 , the image processing device 100 may include an input unit 110, a communication unit 120, a display 130, a memory 140, and a control unit 150.

The input unit 110 generates input data in response to various user inputs and may include various input means. For example, the input unit 110 includes a keyboard, a key pad, a dome switch, a touch panel, a touch key, a touch pad, and a mouse. (mouse), menu button (menu button), etc. may be included, but is not limited thereto.

The communication unit 120 is a component that communicates with other devices. For example, the communication unit 120 may perform 5th generation communication (5G), long term evolution-advanced (LTE-A), long term evolution (LTE), Bluetooth, bluetooth low energy (BLE), near field communication (NFC), Wireless communication such as WiFi communication or wired communication such as cable communication may be performed, but is not limited thereto. For example, the communication unit 120 may receive images, machine learning models, etc. from other devices, and image information processing results (dimensional reduction processing results, additional processing processing results, comparison processing results, etc.), machine learning models, etc. can be sent to In this case, when a machine learning model or the like is received from another device, it may be a case in which another device performs learning on a machine learning model in an image processing method described later.

The display 130 displays various image data on a screen, and may be composed of a non-emissive panel or a light-emitting panel. For example, the display 130 may include a liquid crystal display (LCD), a light emitting diode (LED) display, an organic LED (OLED) display, and a micro electromechanical system (MEMS). mechanical systems) display, or electronic paper (electronic paper) display, etc. may be included, but is not limited thereto. Also, the display 130 may be combined with the input unit 110 and implemented as a touch screen or the like.

The memory 140 stores various types of information necessary for the operation of the image processing device 100 . The stored information may include images, image information processing results (dimensional reduction processing results, additional processing processing results, comparison processing results, etc.), machine learning models, program information related to image processing methods to be described later, etc., but is not limited thereto. not. For example, the memory 140 may be of a hard disk type, a magnetic media type, a compact disc read only memory (CD-ROM), or an optical media type according to its type. ), Sagneto-optical media type, Sultimedia card micro type, flash memory type, read only memory type, or RAM type (random access memory type), etc., but is not limited thereto. In addition, the memory 140 may be a cache, a buffer, a main memory, an auxiliary memory, or a separately provided storage system depending on its purpose/location, but is not limited thereto.

The controller 150 may perform various control operations of the image processing device 100 . That is, the control unit 150 may control the execution of an image processing method to be described later, and the rest of the components of the image processing device 100, that is, the input unit 110, the communication unit 120, the display 130, and the memory 140 etc. can be controlled. For example, the control unit 150 may include, but is not limited to, a processor that is hardware or a process that is software that is executed in the corresponding processor.

3 shows a block diagram of the controller 150 in the image processing device 100 according to an embodiment of the present invention.

The control unit 150 controls the execution of an image processing method according to an embodiment of the present invention, and as shown in FIG. 3, a nonlinear dimension reduction unit 151, a semantic variable extraction unit 152, and a semantic variable conversion unit. (153), a nonlinear dimension expansion unit 154, a learning unit 155, and a comparison unit 156. For example, the nonlinear dimension reduction unit 151, the semantic variable extraction unit 152, the semantic variable conversion unit 153, the nonlinear dimension expansion unit 154, the learning unit 155, and the comparison unit 156 may include a control unit ( 150) or a software process executed by the control unit 150, but is not limited thereto.

Hereinafter, an image processing method according to the present invention will be described in more detail.

4 is a flowchart of an image processing method according to an embodiment of the present invention.

An image processing method according to an embodiment of the present invention is a method of processing image information for comparison between images, and may include S201 and S202 as shown in FIG. 4 .

That is, the nonlinear dimensionality reduction unit 151 may generate q' and s _i 'by performing dimensionality reduction processing (S201). Thereafter, the comparator 152 may indirectly perform a comparison with q for each s _i by using the dimensionally reduced q' and s _i ' (S202).

Specifically, in S201, the nonlinear dimension reduction unit 151 converts the query image information (q) and N search target image information (s 1 to s N ) (hereinafter, s ₁ to s ₁ to s _N ), where N is a natural number equal to or greater than 2. s _N is referred to as s _i ), q' and s _i ' may be generated by performing dimensionality reduction of the data, respectively. In this case, q' and s _i ' represent at least one element of the attribute of the corresponding image. In this case, the attribute may be information about various characteristics included in the image.

For example, information on an object included in an image, information on a smaller detailed object included in the object, and the like may be attributes. For example, if a “human face” is included in the image, information about the object of the human face and detailed characteristics of the object of the human face (skin color, eye color, hair color, gender, age, etc.) Information on detailed objects of the person's face (eyes, nose, lips, ears, forehead, jawline, etc.) may correspond to attributes.

The result output through the nonlinear dimension reduction unit 151 has information whose data dimension is nonlinearly reduced compared to the input.

For example, dimensionally reduced q' and s _i ' may be expressed as a vector having a plurality of elements, which may be referred to as a “representation vector”. The expression vector represents an algebraic space corresponding to the dimension of the vector, which may be referred to as “latent space”. That is, the nonlinear dimension reduction unit 151 may play a role of generating an expression vector or a latent space by receiving an input image (black and white or color) of q or s _i .

In general, an image can be represented by a vector, a 2D matrix, or a 3D matrix, and the present invention does not limit a specific format. For example, a color image with 100 pixels in width, 100 pixels in height, and three RGB color channels can be expressed as a 3D matrix of 100x100x3, and this value is converted into a 2x1 expression vector through the nonlinear dimension reduction unit 151. If it is scaled down, it can be expressed that a 100x100x3 image is encoded (or mapped) in a two-dimensional latent space as a result. Of course, a 100x100x3 image may be expressed as a 3D matrix or a 2D matrix having fewer elements than the expression vector.

Here, the nonlinear dimensionality reduction unit 151 performs dimensionality reduction processing, that is, nonlinear dimensionality reduction. The nonlinear dimension reduction process can be expressed as a nonlinear function called 'g()'. When q is input, a low-dimensional expression vector called q(q) can be output. Of course, g() (hereinafter referred to as “dimensional reduction function”), which is a function for dimensionality reduction processing, is also expressed in the form of g()=g ₁ (g ₂ (g ₃ (... At this time, each subfunction can be a linear or nonlinear function, so finally g() becomes a nonlinear function. That is, g() receives dimensionality reduction q' as input and outputs , s _i as input, it outputs dimensionally reduced s _i '.

In particular, dimensionality reduction processing may be performed using various dimensionality reduction algorithms. For example, the dimension reduction algorithm may be Laplacian eigenmaps, Isomap, Local Linear Embedding (LLE), Stochastic Neighbor Embedding (t-SNE), etc., but is not limited thereto.

In addition, dimension reduction processing may be performed using a machine learning model trained using a machine learning technique such as a deep learning technique. In this case, the learning unit 155 may generate a machine learning model by performing learning according to a machine learning technique. For example, a deep learning technique may be an autoencoder, a variational autoencoder (VAE), a generative adversarial network, or the like, but is not limited thereto.

A machine learning model is a model learned according to a machine learning technique through training data of a pair of input data and output data (data set). That is, the machine learning model includes an input layer into which input data is input, an output layer through which output data is output, and a hidden layer provided between the input layer and the output layer and having a function for the relationship between the input data and the output data. At this time, the hidden layer may also be composed of a plurality of layers. That is, when input data is input to the machine learning model, output data according to a corresponding function may be output.

At this time, the machine learning model is also referred to as a “neural network”. Each layer in the artificial neural network is composed of at least one filter, and each filter has a matrix of weights and a matrix of nodes. That is, each element (pixel or node) in the matrix of the corresponding filter may correspond to a weight value.

For example, an autoencoder is a kind of generative model, and its output may be an image according to its learning method. In this case, the generative model may be defined as a model that outputs data corresponding to a predetermined parameter value with a specific probability distribution. For example, a generative model having a one-dimensional Gaussian probability distribution may generate data having a bell-shaped occurrence distribution centered on an average value. In addition, if there is a generative model expressed as the sum of a plurality of two-dimensional Gaussian functions, the generated data will show a distribution with several peaks.

An autoencoder is usually composed of encoder hidden layers that compress data, decoder hidden layers that decompress data, and a central layer between the encoder and the decoder. At this time, the number of nodes in the center of the hidden layer is smaller than that of the input layer and the output layer. In the case of a symmetric autoencoder in which the shapes of the encoder and decoder are symmetric to each other, the hidden layer has a plurality of layers of 2m + 1 (where m is a natural number greater than or equal to 0), and the central layer corresponds to a layer located at the center of the plurality of layers , each layer contains a number of nodes (ie matrices). Of course, in a general autoencoder, the hidden layer that serves as an encoder and the hidden layer that serves as a decoder do not necessarily have to be symmetrical.

In addition, in the case of the variant autoencoder (VAE), statistical techniques are applied to the inside of the hidden layer, and the latent space is learned continuously and structurally compared to simple autoencoders.

An autoencoder usually has a structure in which dimensionality reduction is performed through encoding and then decoded and expanded to the size of the original dimension. That is, the hidden layer (center layer) of the autoencoder has a reduced number of nodes compared to the input layer and the output layer.

For example, in the case of an autoencoder, if X is input and trained to output X, encoding of X is performed in the hidden layer. The autoencoder learned in this way has a feature of generating an output almost similar to X even if a value of X + delta is input thereafter. Mathematically, it can also be expressed that arbitrary multi-dimensional information is projected onto a multi-dimensional manifold through an autoencoder and the result is output. That is, X + delta means that noise, occluded areas, stains, etc. exist in the image, and if these are input, a clean output X with these removed can be obtained.

That is, when q is input to the learned autoencoder using the same input/output, each node value (ie, weight value) of encoding for q is set (confirmed) in the hidden layer. That is, in the central layer of the hidden layer, a node value related to dimensionality reduction process q' is set. Similarly, when s _i is input to the learned autoencoder using the same input/output, a node value related to the dimensionally reduced s _i 'is set in the central layer of the hidden layer. Expression vectors of q' and s _i ' can be derived by using the matrix of each node value set in this way as it is or after processing it.

Meanwhile, the characteristics of the above-described autoencoder can be similarly applied to a variational autoencoder (VAE) and a generative adversarial network.

The dimension finally reduced by the nonlinear dimension reduction unit 151 is variable, and the smaller the dimension of the expression vector, the higher information compression is performed, but a problem may occur that the original image cannot be properly restored. Information is lost in the process of dimensionality reduction. On the other hand, the more similar the expression vector is to the dimension of the original image, the better the information restoration ability, but almost no compression, making it impossible to extract meaningful attributes (features). This can be seen as a trade-off relationship, and it is desirable to determine the dimension of the expression vector according to trial and error, various types of learning, and application.

The expression vector of q' and s _i ' derived in this way represents at least one element with respect to the attribute of the corresponding image. For example, if q and s _i are images related to “human face”, expression vectors of q’ and s _i ’ are “skin color”, “eye color”, “hair color”, “gender”, “age”, It may contain elements related to attributes such as “eyes”, “nose”, “lips”, “ears”, “forehead”, or “jawline”. That is, if at least one element at a specific position of the expression vectors of q' and s _i 'is an element related to the attribute of “skin color”, when the value of the element is changed, when the expression vector is restored as an image, the original “skin color” ” is subject to change. This attribute extraction function may be performed by the semantic variable extractor 152 .

In this case, the property may have a plurality of detailed properties. As an example, if each attribute is about {age, gender, hairstyle, clothing color, face covering pattern, noise pattern}, then each attribute is age: {teens, 20s, 30s, 40s or older}, Gender: {Male, Female, Unknown}, Clothing Color: {Blue, White, Black, Orange}, Face Covering Pattern: {Mask, Sunglasses, Glasses, Scarf, Hand}, Noise Pattern: {Gaussian Random Noise, salt and pepper noise, burst noise}, etc. However, in the present invention, “attribute” and “detailed attribute” may be unified and used as “attribute” without being separately distinguished.

That is, since the expression vector of q' and s _i 'output through the nonlinear dimension reduction unit 151, which is a key element of the present invention, expresses each attribute of the image as an element value, the semantic level It has the advantage of being able to express information at a high level of

In addition, since the expression vectors of q' and s _i ' have lower-dimensional data than the images of q and s _i , faster comparison processing is possible compared to comparison processing in the image domain having a large size. There is an advantage that the amount of stored data can be reduced.

In this way, as the present invention uses expression vectors of q' and s _i ', which represent information at a semantic level, <removal of contamination elements of query image information (q)>, <contamination existing in the search target set (S) Element removal>, <injection of contamination elements present in query image information (q) into each image information (s ₁ to s _N ) of the search target set (S)>, etc. are possible. A detailed description of these will be described later.

Meanwhile, in S201 or S202, the semantic variable extractor 152 performs a function of extracting semantic variables for each image for q and s _i , and can extract attributes included in each image of q and s _i . In particular, the semantic variable extractor 152 may determine which element of the dimensionally reduced expression vector of q' and s _i ' is related to a specific attribute of the corresponding image.

For example, when a “smiley face” image is input to the learned autoencoder (case 1), a first expression vector may be generated according to the node value setting of the central layer of the hidden layer. Similarly, when a “non-smiling face” image is input to the learned autoencoder (second case), a second expression vector can be generated according to the node value setting of the central layer of the hidden layer. At this time, by comparing the first and second expression vectors, it is possible to determine at which position the value of the element has changed the most, and the identified corresponding element can be determined to be an element related to the attribute of “smiley expression”.

For example, the semantic variable extractor 152 may define attributes such as “human face”, “facial feature points”, “smile” in advance and extract attributes of objects included in the image from the input image. have.

Through the semantic variable extractor 152, each attribute of the expression vector of q' and s _i 'can be grasped, and the identified attribute (eg, common attribute) is later converted into a semantic variable transform unit 153. It can be used for variable conversion processing of . Information analyzed through the semantic variable extractor 152 may be provided to other devices through the communication unit 120, and through this, an external user or program may control a specific variable.

On the other hand, in S202, the semantic variable conversion unit 154 uses each attribute of the expression vector of q' and s _i ' extracted by the semantic variable extraction unit 152 to process the element value related to any one attribute, conversion can be performed.

Specifically, the semantic variable conversion unit 154 may perform the following roles.

(1) If q is contaminated (for example, q is covered by an arbitrary object, damaged, noise is generated, geometric deformation occurs, etc.), the source of contamination can be removed. That is, in the expression vector of q', the corresponding contamination cause can be removed by processing the element value of the attribute related to the contamination. Such contaminant removal can be applied even when a specific element (s _i ) of S is contaminated.

Through this, the present invention can perform <removal of contaminating elements of the query image information (q)>, <removal of contaminating elements present in the search target set (S)>, and the like.

(2) If a predetermined weight is given to element values of expression vectors or a linear combination is performed on two or more different expression vectors, semantic level conversion is possible. For example, the semantic variable conversion unit 154 adds element values of the expression vector corresponding to the attribute of “glasses” to the expression vector of “the face of a person without glasses”, thereby adding “the face of a person wearing glasses”. can be converted into an expression vector corresponding to Also, if <expression vector of “person wearing glasses” - expression vector of “person”> is executed, expression vectors related to “glasses” are output, <expression vector of “person wearing glasses” - expression vector of “glasses”> , it is possible to process such as outputting the expression vector of “person”. Through this, the present invention enables <support for semantic level comparison between q and S> and <support for active variable control at semantic level between q and S>. That is, the latent space by the expression vector can be configured to give a higher level of meaning that humans think.

As an example, assuming that there are four attributes {“normal”, “smiling”, “crying”, “frowning”} in the attribute of facial expression, the nonlinear dimension reduction unit 151 uses a variable autoencoder to Suppose we want to do some reduction processing. At this time, if you collect labeled face photos with facial expressions {“normal face”, “smile face”, “crying face”, “frown face”} and learn each of them to create an expression vector {“normal face” , “Smiley face”, “Crying face”, “Frown face”} expression vectors are obtained, which can be converted into expression vectors {“smiley”, “crying”, “frowning”}. For example, to find the expression vector “smile”, subtract the expression vector of “smile face” from the expression vector of “normal face”, the common attribute “face” disappears, and only the value of the expression vector “smile” remains. . At this time, if the expression vector is not simply added or subtracted from the expression vector, and the element value of the specific attribute is weighted together with the linear combination as described above, variable control at the semantic level may be possible.

(3) Due to the characteristics of (2) described above, the present invention <injection of contamination elements present in query image information (q) into each image information (s ₁ to s _N ) of the search target set (S)>, etc. this is possible That is, when there is a random polluted object in q, it is possible to estimate it and inject it into the element (s _i ) of the search target set (S). Specifically, after finding the expression vector corresponding to the contamination object of q from q' and dimensionally reducing S, the element values of the attribute corresponding to the contamination source are assigned to the expression vectors for each of the images (s _i '). By adding or adjusting the weight, it is possible to inject a contaminant element present in q into each element of the set to be searched (S).

On the other hand, in addition to the above (1) by the semantic variable conversion unit 153, the present invention provides the same effect as the above-mentioned (1) through the following (4) by the nonlinear dimension reduction unit 151, that is, <query image information It is possible to remove contaminants of (q)>, <remove contaminants present in the search target set (S)>, etc.

(4) If q or s _i is contaminated (for example, it is obscured by an object, damaged, noise is generated, geometric deformation occurs, etc.), the cause of contamination is removed through dimensionality reduction processing. You may. In this case, the nonlinear dimensionality reduction unit 151 performs dimensionality reduction processing through a machine learning model such as an autoencoder, a variational autoencoder (VAE), or a generative adversarial network. . In such a machine learning model, since the hidden layer (central layer) has a reduced number of nodes compared to the input layer and the output layer, encoding can be performed through the hidden layer if dimensionality reduction is performed through this. In particular, such a machine learning model may be learned based on various first images similar to q or s _i that are contaminated (provided that the first images are images without corresponding contamination). In this case, if contaminated q or s _i 'is input to the machine learning model, the hidden layer is set to a node value in a state in which the pollutant is removed, so the expression vector of q' or s _i 'corresponding to the state in which the pollutant is removed can create According to this principle, the present invention can perform <removal of contaminating elements of query image information (q)>, <removal of contaminating elements existing in the set to be searched (S)>, etc. during the dimension reduction process.

Meanwhile, in S202, the comparator 156 may directly compare expression vectors q' and s _i ', or compare elements of at least one attribute included in expression vectors q' and s _i '. This comparison may also include a comparison after performing the dimension reduction process of removing the contaminant according to (4) above.

In addition, the comparator 156 may perform additional processing on at least one of q' and s _i ' within the corresponding reduced dimension, and then compare them. At this time, the additional processing treatment may be any one of the above-described (1) to (3).

That is, the additional processing may be to change the element value of q' or s _i ' related to at least one attribute. In addition, the additional processing may be a process of removing at least one of the contaminated attributes of q from q'. Also, the additional processing process may be a process of removing at least one attribute contaminated in any one s _i from each s _i '. Further, the additional processing process may be a process of injecting at least one attribute contaminated in q into each s _i '. In addition, the additional processing process may be a process of controlling variables by selectively changing at least one attribute in q' or s _i '. Since the details of these additional processing treatments have been described in (1) to (3), they will be omitted below.

Meanwhile, in S202, the nonlinear dimension expansion unit 154 may decode q' and s _i '. In addition, the nonlinear dimension expansion unit 154 may decode q' or s _i ' after additional processing within the corresponding reduced dimension.

At this time, “decoding process” is a process of converting q' and s _i ', which had information of a semantic level, back into an image as the dimension is reduced. According to this decoding process, the dimension of the data is expanded. “Dimensional expansion” may mean the opposite of the aforementioned dimensionality reduction.

That is, the non-linear dimension expansion unit 154 can inversely expand the dimensionally reduced expression vector to make an image the size of the original image, a larger image, or a smaller image. To this end, an autoencoder, a variational autoencoder (VAE), or a generative adversarial network may be used, but is not limited thereto.

According to the decoding process of the nonlinear dimension expansion unit 154, the comparator 156 may compare decoding processing results for q' and s _i ' in S202. Also, the comparator 156 may compare results of additional processing and decoding processing for q' or s _i '.

For example, the additional processing for q' may be referred to as q'', and the additional processing for s _i ' may be referred to as s _i ''. At this time, the comparator 156 may compare the decoding process result of q' and the decoding process result of s _i '. In addition, the comparator 156 may compare the decoding process result of q″ and the decoding process result of s _{i ′} . In addition, the comparator 156 may compare the decoding process result of q' with the decoding process result of s _i ''. In addition, the comparator 156 may compare the decoding process result of q'' with the decoding process result of s _i ''.

In summary, in terms of mutual comparison of q and the ith element s _i of S, if nonlinear dimensionality reduction g() is applied to each input image, the expression vectors of g(q) and g(s _i ) (i.e., Expression vectors of q' and s _i ') can be obtained, and indirect comparison of q and s _i can be performed. That is, using a predetermined comparison method defined as similarity(), the comparison result of similarity (g(q), g(s _i )) (ie, similarity(q', s _i '), similarity(q'', s _i '), similarity (q', s _i ''), or similarity (q'', s _i '')) can be obtained.

For reference, the manifold hypothesis exists as a theoretical background for dimension reduction processing. The manifold hypothesis is that data that exist in a high dimension actually exist in a manifold (manifold) of a lower dimension. Metaphorically, in the case of a clockwork, it exists in a three-dimensional space, but if the mainspring is widened, it is formed into a single plane. Similarly, for data represented in high dimensions, their actual distributions can be expressed as manifolds in lower dimensions.

In particular, in the present invention, it is preferable to use a non-linear method for dimension reduction processing. For example, a conventional technology such as the Wiener filter uses a method of establishing a mathematical model of how contamination occurs and inversely transforming it to remove contaminant elements. In the case of the conventional PCA (Principle Component Analysis) method, after finding a new coordinate system describing the input data, the data was processed considering the importance of each dimension. However, these conventional techniques are usually just a method of assuming that the model is a linear model, and in the case of using a non-linear model, a simplification process such as assuming that the model is linear for a specific local section is required. In addition, this simplified model also had a difficulty in re-estimating detailed parameters assuming that they were prior knowledge. Of course, the nonlinear dimension reduction method is theoretically more complicated than the linear model, and the time and amount of computation required for learning may increase. However, using various techniques and structures developed in the latest artificial neural network technology has the advantage of reducing the learning time and amount of computation. For example, such an artificial neural network includes an autoencoder, a variational autoencoder (VAE), a generative adversarial network, and the like, and learning about them is performed through the learning unit 155. can be controlled

5 shows a conceptual diagram for image comparison using nonlinear dimensionality reduction processing according to the present invention.

Referring to FIG. 5, the present invention does not directly calculate the degree of mutual similarity with the elements of S when q is input, but g() corresponding to dimension reduction processing for q and the ith element (s _i ) of S After applying the transformation relationship called, it is characterized in that a comparison called similarity (g (q), g (s _i )) is performed. This dimensionality reduction process can be expressed by the g() operator, and can also be expressed in the form of multiple composition functions such as g()=g ₁ (g ₂ (g ₃ (...). Each function constituting the multiple composition function is It can be linear or non-linear.

The advantage obtained through the dimensionality reduction process is that the dimensions of the data of g(q) and g(s _i ) (that is, the dimensions of the data of q' and s _i ') can be significantly reduced compared to the original input images q and s _i . There is. For example, it is the same expression as expressing a 1024x1024 2D image as a 100x1 vector. However, this idea may be similar to the method used in the conventional PCA (principal component analysis), but PCA is a linear transformation, whereas the dimensionality reduction process according to the present invention uses a non-linear method.

Another advantage obtained through dimensionality reduction processing is that much more diverse meanings can be included in the output expression vector compared to information output through linear transformation such as PCA. As an example, PCA is limited to finding eigenvectors and eigenvalues in consideration of the occurrence distribution of a given dataset, specifically the variance, but uses a non-linear method such as an autoencoder or a variational autoencoder. In this case, a dimensionally reduced expression vector can create a latent space at a more abundant semantic level.

6 shows an example of face comparison according to the present invention.

In FIG. 6, it is assumed that the image q is a “frown face”, and each element of S (S={s ₁ , s ₂ , …s _i , …s _N }, i∈{1, 2, …N}) Assume that the images corresponding to are “frown face” (s ₁ ), “angry face” (s ₂ ), ... “smiley face” (s _i ), ... “sad face” (s _N ), respectively. . When dimension reduction processing according to the present invention is performed, q becomes g(q) (ie, q'), and S is S'={g(s ₁ ), g(s ₂ ), . . . g(s _i ), . . . g(s _N )}. According to the present invention, it is possible to perform a comparison between g(q) and each g(s _i ). In particular, in the latent space described by the expression vector, the expression vector of the attribute corresponding to “facial expression” may be removed from the expression vector “face” and the expression vector of the attribute “smile” may be injected. In this case, for the faces of q and S, facial expressions can be converted into smiles while keeping other factors such as face shape and skin color unchanged as much as possible. In other words, by setting facial expression as a variable to control, the possibility of inconsistency due to facial expression can be reduced when comparing images.

7 shows another example of face comparison according to the present invention.

Referring to FIG. 7, when contamination occurs in the q image (or when contamination occurs in the s _i image), such as when noise is included, when the area around the mouth is covered by a mask, when a part of the image is damaged, etc. The present invention can remove attributes corresponding to the contaminants from q' (or s _i ') after dimensionality reduction processing.

A more detailed description of this contamination situation is as follows. That is, suppose that q is contaminated by an element a, such as q = q + a. In this case, if q is used as it is and compared with s _i , the comparison cannot be performed properly. Accordingly, for accurate comparison, a process of comparing with the elements of S is required after taking measures to remove a, which is a contaminant element. However, in the case of the prior art, q was processed and compared in the image dimension. Here, q = q + a, which is a formula representing the contaminated state, is for simple explanation, and more generally, it can be expressed in an arbitrary function form such as q = f (q). That is, the prior art used the principle of finding the source of contamination a or its function f() and then taking its inverse transformation. However, since processing and comparison must be performed in the image dimension, the amount of calculation and data storage space for calculation increase.

On the other hand, in the present invention, after encoding high-dimensional image data into a low-dimensional image (ie, dimension reduction process), noise is removed from the reduced dimension. This can be supported by the manifold hypothesis as described above, and since random noise in particular has no special rules, it may not be encoded through nonlinear dimensionality reduction processing.

In addition, the processing of (4) described above can be applied. That is, if there is a hidden part on the face (ie, a contamination source exists), the learning of the machine learning model is performed using the covered face and the non-covered face, and dimensionality reduction processing is performed using the trained machine learning model , it is possible to secure an expression vector including a hidden structure behind the occluded part. In other words, if a machine learning model of “nonlinear dimensionality reduction that removes occluded parts” is obtained through learning, and a face with occluded parts is input to the model, the output expression vector is “representation of a face without occluded parts” vector”.

In addition, although not shown in FIG. 7, through the semantic variable conversion unit 154 according to the above-described (1), a contaminant element existing in q may be extracted and injected into each element of S.

However, in FIGS. 6 and 7 , the attribute “facial expression” is described as an example, but the present invention is not limited thereto and can be applied to various attributes, of course.

8 shows an example of comparison after q and s _i are each expressed with a plurality of expression vectors in the present invention.

Referring to FIG. 8, in S201, a plurality of q' for q, that is, g ⁽¹⁾ (q), g ⁽²⁾ (q), ... g ^(k) (q) can be generated respectively. Similarly, for s _i , multiple s _i ′, i.e. g ⁽¹⁾ (s _i ), g ⁽²⁾ (s _i ), . . . g ^(k) (s _i ) can be generated respectively. Here, k is a natural number greater than or equal to 2. In this case, k q' or s _i ' may represent different attributes. That is, k each of s ₁ 'to s _N ' may be generated to correspond to the attribute according to each q'.

These multiple expression vectors may be generated through different dimensionality reduction processing (multiple machine learning models). For example, k pieces of q' and k pieces of s _i ' may be respectively generated by using k machine learning processes for dimensionality reduction while processing different attributes. In addition, expression vectors output through one dimensionality reduction process (machine learning model) may be generated through various linear combinations.

Thereafter, in S202, comparison may be performed using a plurality of q' and s _i '. Of course, a plurality of q' and s _i ' may be compared after additional processing, or may be compared after decoding. The plurality of expression vectors generated in this way provide a new meaning that did not exist in the originally input image, and the comparison in S202 can be more elaborate and rich.

For example, even though the input q was simply a “side face”, using the above method, the images of q and S are expanded into a plurality of expression vectors, respectively, {“side face”, “front face”, “from above It can be extended to a number of expression vectors corresponding to angles, such as “face seen”, “face seen from below”}. In addition, even if q was simply “face in their twenties”, the faces of q and S were extended expressions such as {“teenage face”, “large face”, “large face”, “large face”, “large face”} By creating multiple vectors each, comparison can be performed by creating various independent variables rather than simple comparison. In this case, applications such as finding a person who has not been found for decades or performing comparison by transforming image information captured in space A into images taken in other spaces such as {B, C, D, E} this is possible

In addition, referring to FIG. 8, since images of S in addition to q can extend such an expression vector, clues that could not be found in simple comparison can be found. For example, a direct comparison of q and s _i may not reveal significant similarities. However, g ⁽¹⁾ (q), which is dimensionally reduced to convert q into the first variable, and g ⁽³⁾ (s _i ), which is dimensionally reduced to transform s _i into a third variable, can be similar. have. In this case, the relationship between the first variable and the third variable can be grasped to infer the correlation and causal relationship between the events.

The additional processing in S201 and S202 and the processing in (1) to (4) described above can be used as a kind of image preprocessing technique performed before mutual comparison of given image information. This can be used for various services such as search or data purification.

In summary, the present invention has a difference from the prior art in the following points.

<Resolving the problem when q or s _i is contaminated>

Even if the image information of q or s _i is contaminated (occluded by an arbitrary object, noise is generated, damaged, or captured too small), the present invention provides the dimension of the nonlinear dimension reduction unit 151 Such a contaminating factor may be removed through a reduction process or through additional processing of the semantic variable conversion unit 153 in the corresponding dimension for dimensionally reduced q' or s _i '. In particular, through the additional processing of the variable conversion unit 153 When the contaminant element exists only in q, the comparison inconsistency problem caused by the contaminant element can be offset by identifying the contaminant element of q and injecting it into _si .

<Semantic Level Comparison Support>

The present invention encodes input video information into semantic information through dimensionality reduction processing, and then i) uses the encoded information as it is, ii) weights element values of specific attributes in the encoded information, or iii) encodes A predetermined conversion may be performed on the information, or iv) it may be converted into an image domain through a decoding process of dimensionally extending the information after processing any one of i) to iii). Thereafter, the present invention performs mutual comparison using the result processed by any one of i) to iv) described above. At this time, the encoded information includes semantic information of a high level that ordinary people think with common sense. By directly or indirectly using such encoded information in a comparison process, a semantic level comparison is possible, and an indirect comparison of q and s _i is possible.

<Possible to actively control variables at the semantic level>

In addition, the present invention supports variable control at the semantic level. At this time, variable control is a methodology commonly used in scientific research, and can also be used for comparison between images. In particular, the present invention has a great difference in providing a variable control function at a semantic level through a process of actively transforming an image.

That is, conventionally, these variables were controlled in advance in the step of preparing q and S. For example, when S is “a set of faces of people whose gender is not distinguished” and this is defined as a whole set, a subset from which only “female faces” are separately extracted can be defined as S _f . At this time, in the prior art, in order to mutually compare the images of the subset of S _f with q, the comparison is performed after applying variable control to q and S _f in the same image dimension.

On the other hand, in the present invention, while dimensionally reducing images corresponding to elements of q or S, “Match with smiling face”, “Match with sad face”, “Match with average Korean face”, “Glasses” “Matches the faces of people wearing glasses”, “Matches the faces of people not wearing glasses”, “Matches the faces of people wearing masks”, “Matches the faces of people not wearing masks”, “It says they were filmed with a specific camera It is possible to control variables at the semantic level by actively transforming, such as “matching by converting like a photograph at the time of assumption”, “matching the video style to look like a picture drawn by Hong Gil-dong”.

That is, conventionally, large features of elements within a given set S are identified, and detailed attributes of each feature are identified. After that, if each attribute is classified as a variable, the existing elements are classified, a subset is formed, and the variable is controlled. In the present invention, the attributes of the elements are selectively transformed and processed in a dimensionally reduced state, can do.

In particular, in the present invention, a learning process of a machine learning model in which attributes are defined in advance and dimensionality reduction processing is performed accordingly can be used for variable control. In addition, the present invention can perform variable control by performing processing (arithmetic processing) such as linear combination of expression vectors describing the latent space corresponding to each attribute.

The present invention configured as described above provides an image comparison processing technology between a query input image and a search target set, but provides a new method that can control variables at the semantic level by actively and selectively changing high-level feature elements present in the image. There is an advantage that image comparison processing is possible. In addition, the image comparison processing according to the present invention efficiently performs calculation processing through data dimension reduction, enabling faster comparison processing compared to comparison processing in the image domain having a large size and reducing the amount of stored data. There is an advantage. In addition, the present invention has an advantage that it can be used in various fields that require image search, similarity comparison, classification, and the like.

In the detailed description of the present invention, specific embodiments have been described, but various modifications are possible without departing from the scope of the present invention. Therefore, the scope of the present invention is not limited to the described embodiments, and should be defined by the following claims and equivalents thereof.

100: image information processing device 110: input unit
120: communication unit 120: communication unit
130: display 140: memory
150: input unit 151: nonlinear dimension reduction unit
152; Semantic variable extraction unit 153: semantic variable conversion unit
154: non-linear dimension expansion unit 154: learning unit
155: comparison unit

Claims (17)

An image processing method in which each step is performed by a computing device,
Non-linear data for each query image information (q) and search target image information (s ₁ to s _N ) of N pieces (N is a natural number of 2 or more) (hereinafter, s ₁ to s _N are referred to as s _i ) a generation step of performing dimensionality reduction to generate q' and s _i ' representing attributes of a corresponding image with at least one element; and
A comparison step of indirectly performing a comparison with q for each s _i using dimensionally reduced q ' and s _i '; including,
The q' and _si ' are generated using a pre-learned model using a machine learning technique,
The model includes an input layer through which input data is input, an output layer through which output data is output, and at least one hidden layer connecting the input layer and the output layer, the number of nodes being reduced compared to the input layer and the output layer, respectively.
Wherein q' and s _i ' are generated based on at least one hidden layer in which node values are set by inputting q and s _i to the model, respectively.
An image processing method in which each step is performed by a computing device,
Non-linear data for each query image information (q) and search target image information (s ₁ to s _N ) of N pieces (N is a natural number of 2 or more) (hereinafter, s ₁ to s _N are referred to as s _i ) a generation step of performing dimensionality reduction to generate q' and s _i ' representing attributes of a corresponding image with at least one element; and
A comparison step of indirectly performing a comparison with q for each s _i using dimensionally reduced q ' and s _i '; including,
In the generating step, at least one attribute contaminated in q is removed from q' or at least one attribute contaminated in any one s _i is removed from q' through a pre-learned model using a machine learning technique. and performing the data dimensionality reduction while removing _i '.
According to claim 1 or 2,
The generating step includes generating a plurality of q' representing different attributes, and generating a plurality of each of s ₁ ' to s _N ' to correspond to the attribute according to each q'.
According to claim 1 or 2,
Wherein the comparing step directly compares q' and s _i 'or compares elements of at least one attribute included in q' and s _i '.
According to claim 1 or 2,
Wherein the comparing step includes comparing at least one of q' and s _i ' after additionally processing it within a corresponding reduction dimension.
According to claim 5,
The additional processing comprises changing an element value of q' or s _i ' related to at least one attribute.
According to claim 5,
Further comprising an extraction step of extracting information about attributes of q and s _i ,
Wherein the comparing step comprises additionally processing at least one of q' and s _i ' within a corresponding reduction dimension using the information extracted in the extracting step.
According to claim 5,
The additional processing includes removing at least one attribute contaminated from q from q', or removing at least one attribute contaminated from one s _i from the corresponding s _i '.
According to claim 5,
Wherein the additional processing comprises injecting at least one of the contaminated attributes in q into each s _i '.
According to claim 1 or 2,
The comparison step is an image processing method comprising the step of controlling variables by selectively changing attributes of q' or s _i '.
According to claim 1 or 2,
The comparing step comprises decoding and comparing q' and s _i ', or decoding and comparing q' or s _i ' after additional processing within a corresponding reduction dimension.
A memory storing query image information (q) and N (where N is a natural number of 2 or more) search target image information (s ₁ to s _N ) (hereinafter, s ₁ to s _N are referred to as s _i ); and
A control unit controlling a comparison with q for each s _i using q and s _i stored in the memory; includes,
The control unit,
Non-linear data dimension reduction is performed on q and s _i , respectively, to generate q' and s _i ', which represent at least one element for the attribute of the image, and using the dimensionally reduced q' and s _i ' Indirectly performing a comparison with q for each s _i ,
The q' and _si ' are generated using a pre-learned model using a machine learning technique,
The model includes an input layer through which input data is input, an output layer through which output data is output, and at least one hidden layer connecting the input layer and the output layer, the number of nodes being reduced compared to the input layer and the output layer, respectively.
The q' and s _i ' are generated based on at least one hidden layer in which node values are set by inputting q and s _i to the model, respectively.
A memory storing query image information (q) and N (where N is a natural number of 2 or more) search target image information (s ₁ to s _N ) (hereinafter, s ₁ to s _N are referred to as s _i ); and
A control unit controlling a comparison with q for each s _i using q and s _i stored in the memory; includes,
The control unit,
Non-linear data dimension reduction is performed on q and s _i , respectively, to generate q' and s _i ', which represent at least one element for the attribute of the image, and using the dimensionally reduced q' and s _i ' Indirectly performing a comparison with q for each s _i ,
When the q' and s _i ' are generated, at least one attribute contaminated in the q is removed from q' or contaminated in any one s _i through a model pre-learned by a machine learning technique. An image processing device that performs the data dimensionality reduction while removing at least one attribute from the corresponding s _{i '} .
Receiving query image information (q) and N pieces (where N is a natural number of 2 or more) of search target image information (s ₁ to s _N ) (hereinafter, s ₁ to s _N are referred to as s _i ) from the first device one communications department; and
A control unit for controlling a comparison with q for each s _{i using q and s i received} by the communication unit, and controlling to transmit the comparison result to the first device or the second device; includes,
The control unit,
Non-linear data dimension reduction is performed on q and s _i , respectively, to generate q' and s _i ', which represent at least one element for the attribute of the image, and using the dimensionally reduced q' and s _i ' Indirectly performing a comparison with q for each s _i ,
The q' and _si ' are generated using a pre-learned model using a machine learning technique,
The model includes an input layer through which input data is input, an output layer through which output data is output, and at least one hidden layer connecting the input layer and the output layer, the number of nodes being reduced compared to the input layer and the output layer, respectively.
The q' and s _i ' are generated based on at least one hidden layer in which node values are set by inputting q and s _i to the model, respectively.
Receiving query image information (q) and N pieces (where N is a natural number of 2 or more) of search target image information (s ₁ to s _N ) (hereinafter, s ₁ to s _N are referred to as s _i ) from the first device one communications department; and
A control unit for controlling a comparison with q for each s _{i using q and s i received} by the communication unit, and controlling to transmit the comparison result to the first device or the second device; includes,
The control unit,
Non-linear data dimension reduction is performed on q and s _i , respectively, to generate q' and s _i ', which represent at least one element for the attribute of the image, and using the dimensionally reduced q' and s _i ' Indirectly performing a comparison with q for each s _i ,
When the q' and s _i ' are generated, at least one attribute contaminated in the q is removed from q' or contaminated in any one s _i through a model pre-learned by a machine learning technique. An image processing device that performs the data dimensionality reduction while removing at least one attribute from the corresponding s _{i '} .
A memory storing query image information (q) and N (where N is a natural number of 2 or more) search target image information (s ₁ to s _N ) (hereinafter, s ₁ to s _N are referred to as s _i );
a non-linear dimensional conversion unit for performing non-linear data dimensionality reduction on q and s _i , respectively, to generate q' and s _i ' representing at least one element with respect to an attribute of a corresponding image;
A comparison unit for performing a comparison with q for each s _i using q 'and s _i '; includes,
The q' and _si ' are generated using a pre-learned model using a machine learning technique,
The model includes an input layer through which input data is input, an output layer through which output data is output, and at least one hidden layer connecting the input layer and the output layer, the number of nodes being reduced compared to the input layer and the output layer, respectively.
The q' and s _i ' are generated based on at least one hidden layer in which node values are set by inputting q and s _i to the model, respectively.
A memory storing query image information (q) and N (where N is a natural number of 2 or more) search target image information (s ₁ to s _N ) (hereinafter, s ₁ to s _N are referred to as s _i );
a non-linear dimensional conversion unit for performing non-linear data dimensionality reduction on q and s _i , respectively, to generate q' and s _i ' representing at least one element with respect to an attribute of a corresponding image;
A comparison unit for performing a comparison with q for each s _i using q 'and s _i '; includes,
When generating q' and s _i ', the nonlinear dimension conversion unit removes at least one attribute contaminated from q' from q' through a pre-learned model using a machine learning technique, or An image processing device that performs the data dimensionality reduction while removing at least one of the contaminated attributes of s _i from the corresponding s _i '.

Images