KR20240009862A - A method for detecting a deepfake image and an electronic device for the method - Google Patents

Abstract

The present invention provides a deepfake image detecting method which comprises the following steps of: coupling a patch embedding function based on a segmented face image and a CNN function based on a convolutional neural network; inputting a class token learned by a true label, a distillation token based on learning through comparison of a teacher network inference result with the true label, and a global pooling function to which the patch embedding function and the CNN function are coupled into a transformer; and performing machine learning for detecting a deepfake image based on a loss function outputted by the transformer.

Description

Deepfake image detection method and electronic device therefor {A METHOD FOR DETECTING A DEEPFAKE IMAGE AND AN ELECTRONIC DEVICE FOR THE METHOD}

The present invention provides a deep fake image detection method that can distinguish fake images from real images through machine learning and an electronic device therefor.

Damage caused by forged images, such as deepfake images, is increasing rapidly with the advent of the Internet of Things era. The victims range widely from celebrities to the general public, and forged and altered images are used for various crimes such as digital sex crimes, illegal copying, and copyright infringement. Furthermore, damage caused by forged videos (e.g., deepfake videos), in which images are falsified frame by frame, is also increasing with the development of computing devices.

There is a problem in that forged images and forged videos produced using advanced artificial intelligence technology are difficult to determine whether they have been forged or altered with the human eye. In addition, these forgery or falsification technologies have a problem that is rapidly evolving along with the development of artificial intelligence technology.

Therefore, in order to prevent damage caused by forged images and forged videos, technology for determining forged images and forged videos is required.

The present invention provides a method for improving the accuracy of deepfake image detection to prevent damage caused by forged images and forged videos.

The present invention includes the steps of combining a CNN function based on a convolutional neural network, class tokens learned with true labels, and comparing the teacher network inference results with the true labels. Inputting a distillation token based on learning through and a global pooling function combining the patching embedding function and the CNN function into a transformer, and a loss function output by the transformer. Provides a deepfake image detection method that includes performing machine learning to detect deepfake images based on a function).

According to one embodiment, the deepfake image detection method may include extracting a face image from an image using a multitask cascaded convolutional network (MTCNN) model and setting a specific region for segmenting the image from the face image. You can.

According to one embodiment, the patch embedding function is determined based on the patch, which is a segmented image, and the exponential number of the segmented patches, and the CNN function is determined based on the number of CNN features. You can.

According to one embodiment, the global pooling function may be determined by concatenating the embedding function and the CNN function for each channel.

According to one embodiment, the loss function is determined based on a loss function for a fake image and a loss function for a real image, and the step of performing the machine learning is to perform machine learning by setting a normalization weight with the loss function. can do.

According to one embodiment, the loss function for the fake image is determined based on the odds ratio of the teacher network, the odds ratio of the class token, and the odds ratio of the distillation token for the fake image, and the loss function for the real image is determined based on the odds ratio of the teacher network and the odds ratio of the distillation token. The loss function may be determined based on the odds ratio of the teacher network, the odds ratio of the class token, and the odds ratio of the distillation token for the real image.

The present invention is an operation unit that combines a patch embedding function based on a segmented face image and a CNN function based on a convolutional neural network, and a class token learned with a true label. , a transformer that receives a distillation token based on learning through comparison of the teacher network inference result and the actual label, and a global pooling function combining the patching embedding function and the CNN function. ) and a machine learning performing unit that performs machine learning to detect deepfake images based on a loss function output by the converter.

According to one embodiment, the electronic device may further include a preprocessor that extracts a face image from an image using a multitask cascaded convolutional network (MTCNN) model and sets a specific region for segmenting the image from the face image.

According to one embodiment, the patch embedding function is determined based on the patch, which is a segmented image, and the exponential number of the segmented patches, and the CNN function is determined based on the number of CNN features. You can.

According to one embodiment, the global pooling function may be determined by concatenating the embedding function and the CNN function for each channel.

According to one embodiment, the loss function is determined based on a loss function for a fake image and a loss function for a real image, and the execution unit may perform machine learning by setting a normalization weight with the loss function.

According to one embodiment, the loss function for the fake image is determined based on the odds ratio of the teacher network, the odds ratio of the class token, and the odds ratio of the distillation token for the fake image, and the loss function for the real image is determined based on the odds ratio of the teacher network and the odds ratio of the distillation token. The loss function may be determined based on the odds ratio of the teacher network, the odds ratio of the class token, and the odds ratio of the distillation token for the real image.

The present invention provides a method to improve the accuracy of deepfake image detection by applying both local and global features of the distillation token and extracted image to a learning model for deepfake detection.

1 is a diagram illustrating the deepfake image detection process disclosed in the present invention.
Figure 2 is a flowchart illustrating the deepfake image detection method disclosed in the present invention.
Figure 3 is a diagram for explaining the process of deriving the global pooling function disclosed in the present invention.
Figure 4 is a diagram for explaining the loss function derivation process disclosed in the present invention.
Figure 5 is a graph comparing the loss function when using a distillation token according to an embodiment disclosed in the present invention and when according to the prior art.
Figure 6 is a graph comparing the loss function for a fake image and the loss function for a real image when using the deepfake detection algorithm according to an embodiment disclosed in the present invention and when using the prior art.
Figure 7 is a block diagram of an electronic device according to an embodiment disclosed in the present invention.

Since the present invention can make various changes and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all changes, equivalents, and substitutes included in the spirit and technical scope of the present invention. While describing each drawing, similar reference numerals are used for similar components.

Terms such as first, second, A, and B may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another. For example, a first component may be named a second component, and similarly, the second component may also be named a first component without departing from the scope of the present invention. The term and/or includes any of a plurality of related stated items or a combination of a plurality of related stated items.

When a component is said to be "connected" or "connected" to another component, it is understood that it may be directly connected to or connected to the other component, but that other components may exist in between. It should be. On the other hand, when it is mentioned that a component is “directly connected” or “directly connected” to another component, it should be understood that there are no other components in between.

The terms used in this application are only used to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the technical field to which the present invention pertains. Terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and unless explicitly defined in the present application, should not be interpreted in an ideal or excessively formal sense. No. Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the attached drawings.

1 is a diagram illustrating the deepfake image detection process disclosed in the present invention.

According to one embodiment, the deepfake image detection process includes a process of setting a specific area in the image to perform deepfake analysis, a process of preprocessing the image of the set specific area to augment the data, and machine learning of the augmented data. It may include a process of performing learning by applying it to a model, and a deepfake detection process of determining whether the detected image is a real image or a fake image based on the learning results.

According to one embodiment, the machine learning model for detecting deepfake images may use a vision transformer (ViT) model and a convolutional neural network (CNN) model. According to various embodiments, the ViT model assigns an order to the patches that segment the image, and the location information of the patches within the image can be connected to a multi-head self attention layer (MSL). Therefore, when using the ViT model, it may be possible to learn relationships between different parts within an image.

According to one embodiment, the MSL of the ViT model can embed global information about the entire image. According to various embodiments, the ViT model may contain more global information than the ResNet of the lower layer.

According to one embodiment, the CNN model includes information about pixel values around the image, so it is possible to detect the phenomenon in which the surrounding pixels become unnatural due to image compositing. According to various embodiments, local information and global spatial information can be obtained by combining patch embeddings according to a CNN model and a ViT model.

Figure 2 is a flowchart illustrating the deepfake image detection method disclosed in the present invention. According to one embodiment, the flowchart shown in FIG. 2 may be performed by the electronic device shown in FIG. 7.

According to one embodiment, a face image may be extracted from an image using a multitask cascaded convolutional network (MTCNN) model in step S210. According to various embodiments, a specific area for segmenting the image may be set in the face image extracted through steps S220 to S210.

According to one embodiment, a landmark can be created from the extracted face image and the difference in facial structural identity between the real image and the fake image can be extracted. According to various embodiments, the reason for generating landmarks from extracted face images is to more generalize the model by cutting out part of the face image. Steps S210 and S220 may be included in the preprocessing process shown in FIG. 1, and this preprocessing process can prevent overfitting of the learning model.

According to one embodiment, a patch embedding function based on the segmented face image and a CNN function may be combined in step S230. According to various embodiments, the face image can be segmented into desired patch sizes and the patches can be embedded in one CNN layer using the patch size kernel.

According to one embodiment, the patch embedding function can determine the features of the face patch, and the CNN function can determine the overall features of the face image. According to the present invention, both the patch embedding function and the CNN function are considered in machine learning, so deepfake image detection performance can be improved compared to a machine learning model in which only the patching embedding function is considered. A more detailed description of step S230 will be described later through the description of FIG. 3.

According to one embodiment, a class token learned as a true label in step S240, a distillation token based on learning through comparison of the teacher network inference result and the actual label, and patching A global pooling function that combines an embedding function and a CNN function can be input to the transformer. According to various embodiments, the features of the image input to the converter may interact for each patch, and the CNN function may interact with surrounding features of the image.

According to one embodiment, the teacher network may refer to a structure that makes predictions based on an existing well-trained network and learns a student network by comparing this with an actual label. According to various embodiments, class tokens may be learned from actual labels and distillation tokens may be learned based on predictions of the teacher network. A more detailed description of step S240 will be described later through the description of FIG. 4.

According to one embodiment, machine learning to detect deepfake images can be performed based on the loss function output by the converter in step S250. According to various embodiments, machine learning may be performed to minimize the loss function.

According to one embodiment, deepfake images can be detected in step S260 through a machine learning model learned through step S250.

Figure 3 is a diagram for explaining the process of deriving the global pooling function disclosed in the present invention. According to one embodiment, the derivation process shown in FIG. 3 may be performed by the electronic device shown in FIG. 7.

According to one embodiment, the patch embedding function may be defined as Equation 1 below.

[Formula 1]

In Equation 1, Z _p is the patching embedding function, x _p is the patch, E is the learnable embedding, and N is the exponential nuber of the divided patches.

According to various embodiments, the CNN function can be defined as Equation 2 below.

[Formula 2]

In Equation 2 above, Z _f is a CNN function, x _f is a segmented image through an efficient net, and M is the number of CNN features.

According to one embodiment, the CNN function and the patching embedding function can be concatenated for each channel, and the result of concatenating the CNN function and the patching embedding function ( ) can be applied to global pooling.

Figure 4 is a diagram for explaining the loss function derivation process disclosed in the present invention. According to one embodiment, the derivation process shown in FIG. 4 may be performed by the electronic device shown in FIG. 7.

According to one embodiment, the class token, the distillation token, and the result through global pooling ( ) can be entered. According to various embodiments =Global pooling( ) may be an input vector of (N+M, N) input to the converter.

According to one embodiment, the final input value input to the converter may be defined as Equation 3 below.

[Formula 3]

In Equation 3 above, Z ₀ is the final input value, x _class is a class token, is the result of global pooling, x _distilaation is the distillation token, and E _pos is the learnable position embedding value.

According to various embodiments, the final loss function may be determined based on the loss function for the real image and the loss function for the fake image, and more specifically, may be determined through Equation 4 below.

[Formula 4]

In Equation 4 above, is the final loss function, is the loss function for the fake image, is the loss function for the actual image.

According to one embodiment, the loss function for the fake image can be defined as Equation 5 below.

[Formula 5]

In Equation 5 above, is the loss function for the fake image, λ is a variable that can be set by the user, is the binary cross entropy loss, is the logit of the teacher model for fake prediction, is the logit of the distillation token for fake prediction, is the logit of the class token for fake prediction, and y and σ are sigma functions.

According to various embodiments, the loss function for the actual image may be defined as Equation 6 below.

[Formula 6]

In Equation 6 above, is the loss function for the actual image, λ is a variable that can be set by the user, is the binary cross entropy loss, is the logit of the teacher model for fact prediction, is the logit of the distillation token for fact prediction, is actually the logit of the class token for prediction, and y and σ are sigma functions.

Figure 5 is a graph comparing the loss function when using a distillation token according to an embodiment disclosed in the present invention and when according to the prior art.

According to the graph shown in Figure 5, it can be seen that as the weight for fake loss increases, the deviation in loss increases, and it can be seen that the loss when distillation tokens are applied is relatively less than when distillation tokens are not applied. It can be seen that the performance of the deepfake detection learning model is improved when distillation tokens are applied.

Figure 6 is a graph comparing the loss function for a fake image and the loss function for a real image when using the deepfake detection algorithm according to an embodiment disclosed in the present invention and when using the prior art.

According to the graph shown in FIG. 6, the loss function for the fake image when using the learning model proposed in the present invention and the loss function for the real image are compared to the loss function for the fake image when using the conventional SOTA model. It can be seen that the loss function is smaller than the loss function for the actual image, and it can be seen that the accuracy of deepfake detection is improved compared to the prior art when the learning model proposed in the present invention is followed.

Figure 7 is a block diagram of an electronic device according to an embodiment disclosed in the present invention.

According to one embodiment, the electronic device 700 extracts a face image from an image using the MTCNN model, a preprocessor 710 that sets a specific area for segmenting the image from the face image, and a patch based on the segmented face image. An operation unit 720 that combines an embedding function and a CNN function based on a convolutional neural network, a class token learned from an actual label, a distillation token and a patching embedding function based on learning through comparison of the teacher network inference result and the actual label, and a CNN function. A converter 730 that receives a global pooling function combined with; and a machine learning performing unit 740 that performs machine learning to detect deepfake images based on the loss function output by the converter 730.

The above description is merely an illustrative explanation of the technical idea of the present invention, and various modifications and variations will be possible to those skilled in the art without departing from the essential characteristics of the present invention. Accordingly, the embodiments implemented in the present invention are not intended to limit the technical idea of the present invention, but rather to explain it, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be interpreted in accordance with the claims below, and all technical ideas within the equivalent scope should be construed as being included in the scope of rights of the present invention.

Claims (12)

Combining a patch embedding function based on a segmented face image and a CNN function based on a convolutional neural network;
A class token learned with a true label, a distillation token based on learning through comparison of the teacher network inference result with the actual label, the patching embedding function, and the CNN function. Inputting the combined global pooling function into a transformer; and
Including performing machine learning to detect deepfake images based on the loss function output by the converter,
How to detect deepfake images. According to paragraph 1,
Extracting a face image from an image using a multitask cascaded convolutional network (MTCNN) model; and
Including setting a specific area for segmenting the image in the face image,
How to detect deepfake images. According to paragraph 1,
The patch embedding function is determined based on the patch, which is a segmented image, and the exponential number of the segmented patches, and the CNN function is determined based on the number of CNN features.
How to detect deepfake images. According to paragraph 1,
Characterized in that the global pooling function is determined by concatenating the embedding function and the CNN function for each channel,
How to detect deepfake images. According to paragraph 1,
The loss function is determined based on the loss function for the fake image and the loss function for the real image,
The step of performing machine learning is characterized in that machine learning is performed by setting normalization weights with the loss function.
How to detect deepfake images. According to clause 5,
The loss function for the fake image is determined based on the odds ratio of the teacher network, the odds ratio of the class token, and the odds ratio of the distillation token for the fake image, and the loss function for the real image is determined based on the odds ratio of the teacher network for the fake image. Characterized in that it is determined based on the odds ratio of the teacher network, the odds ratio of the class token, and the odds ratio of the distillation token.
How to detect deepfake images. An operation unit that combines a patch embedding function based on a segmented face image and a CNN function based on a convolutional neural network;
A class token learned with a true label, a distillation token based on learning through comparison of the teacher network inference result with the actual label, the patching embedding function, and the CNN function. A transformer that receives the combined global pooling function as input; and
Comprising a machine learning performance unit that performs machine learning to detect deepfake images based on the loss function output by the converter,
Electronic devices. In clause 7,
Further comprising a preprocessor that extracts a face image from the image using a multitask cascaded convolutional network (MTCNN) model and sets a specific region for segmenting the image in the face image,
Electronic devices. In clause 7,
The patch embedding function is determined based on the patch, which is a segmented image, and the exponential number of the segmented patches, and the CNN function is determined based on the number of CNN features.
Electronic devices. In clause 7,
Characterized in that the global pooling function is determined by concatenating the embedding function and the CNN function for each channel,
Electronic devices. In clause 7,
The loss function is determined based on a loss function for the fake image and a loss function for the real image, and the execution unit performs machine learning by setting a normalization weight with the loss function.
Electronic devices. According to clause 11,
The loss function for the fake image is determined based on the odds ratio of the teacher network, the odds ratio of the class token, and the odds ratio of the distillation token for the fake image, and the loss function for the real image is determined based on the odds ratio of the teacher network for the fake image. Characterized in that it is determined based on the odds ratio of the teacher network, the odds ratio of the class token, and the odds ratio of the distillation token.
Electronic devices.

Images