JP2024154761A - Adversarial attack countermeasure support system, method, and program - Google Patents

Abstract

To support measures against adversarial attacks with various attack patterns for an inference model.SOLUTION: A system for supporting measures against adversarial attacks includes: an attack encoder which encodes attack embedding from an original image; a noise adding unit which adds random noise to the attack embedding output from the attack encoder; a generator which generates an attack image to attack an inference model using the attack embedding with the random noise added thereto; an inference unit which performs inference on the attack image using the inference model, to obtain an inference result; a distinction unit which distinguishes the attack image from the original image to calculate a distinction result; a loss calculation unit which calculates a training loss which is a loss for training the attack image on the basis of the inference result and the distinction result; and a parameter update unit which updates the attack encoder, the generator, and the distinction unit on the basis of the training loss.SELECTED DRAWING: Figure 2

Description

This disclosure relates to technology that helps counter hostile attacks.

In recent years, since the emergence of deep neural networks (DNNs), applications using AI (Artificial Intelligence) have increased dramatically. Hereinafter, applications using AI are also referred to as AI applications. AI models used in AI applications perform well when normal data is input, but are vulnerable to adversarial attacks and may cause misclassification or false positives. Therefore, it is important to evaluate the robustness against adversarial attacks before releasing AI applications in actual applications, especially in fields that require high security such as autonomous driving and surveillance systems.

Non-Patent Document 1 discloses a method for evaluating the robustness of an AI model that classifies or detects images. The method disclosed in Non-Patent Document 1 generates adversarial samples by adding noise that cannot be recognized by humans to images using a parameter-free strategy, performs classification or detection on the adversarial samples using the AI model to be evaluated, and evaluates the robustness of the AI model based on the accuracy of the classification or detection.

Patent document 1 describes a method for generating adversarial samples using a Generative Adversarial Network (GAN). The method described in patent document 1 generates adversarial samples that involve visual changes.

International Publication No. WO2020/165935

Reliable Evaluation of Adversarial Robustness with an Ensemble of Diverse Parameter-free Attacks, Francesco Croce and Matthias Hein, ICML 2020

In evaluating the robustness of an AI model, it is preferable to use adversarial samples with various types of features, regardless of whether they can be recognized by the human eye. However, the method disclosed in Non-Patent Document 1 generates adversarial samples whose visual features differ relatively little from the original image, and uses them to evaluate the robustness of the AI model. Therefore, it is not possible to properly evaluate the robustness of an AI model against attacks using images that clearly differ in visual features from the original image but do not appear unnatural to humans.

Patent Document 1 describes a method for generating adversarial samples that have visual characteristics different from those of the original image. The method described in Patent Document 1 has the potential to overcome the lack of attack patterns in the method disclosed in Non-Patent Document 1 as described above. However, the method in Patent Document 1 tends to generate adversarial samples with the simplest attack pattern that can be successfully attacked, and does not generate adversarial samples with a variety of attack patterns.

One objective of this disclosure is to provide technology that helps counter adversarial attacks with a variety of attack patterns on AI models.

An adversarial attack countermeasure support system according to one aspect of the present disclosure includes an attack encoder that encodes an attack embedding from an original image, a noise addition unit that adds random noise to the attack embedding output from the attack encoder, a generator that generates an attack image for attacking an inference model using the attack embedding to which the random noise has been added, an inference unit that performs inference on the attack image using the inference model and obtains an inference result, a discrimination unit that discriminates between the attack image and the original image and calculates the discrimination result, a loss calculation unit that calculates a training loss, which is a loss for training the attack image based on the inference result and the discrimination result, and a parameter update unit that updates the attack encoder, the generator, and the discrimination unit based on the training loss.

One aspect of the present disclosure makes it possible to assist in evaluating the robustness of an AI model against attack images with a variety of attack patterns.

FIG. 2 is a block diagram of a computer system according to the present embodiment. FIG. 2 is a block diagram of an attack generation model learning unit. FIG. 2 is a block diagram of an attack image generation unit. FIG. 1 is a block diagram of an AI model robustness evaluation unit. 13 is a flowchart showing the operation of the AI model robustness evaluation unit.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the present invention is not limited to this embodiment. In addition, in the description of the drawings, the same parts are given the same reference numerals and the description will be omitted as appropriate.

<Hardware configuration>
The apparatus used in the present embodiment may be realized by applying a software program to any suitable computer system.

Figure 1 is a block diagram of a computer system in this embodiment.

The computer system 300 has, as its main components, one or more processors 302, memory 304, a terminal interface 312, a storage interface 314, an I/O (input/output) device interface 316, and a network interface 318. These components are interconnected via a memory bus 306, an I/O bus 308, a bus interface unit 309, and an I/O bus interface unit 310.

Processor 302 may include one or more general purpose programmable central processing units (CPUs) 302A and 302B. For example, computer system 300 may include multiple processors. As another example, computer system 300 may include a single CPU. Processor 302 is a device that executes instructions stored in memory 304 and may include an on-board cache (not shown).

The memory 304 may include a randomly accessible semiconductor memory, a storage device, a volatile storage medium, or a non-volatile storage medium for storing data and programs. The memory 304 may store all or part of the software programs, software modules, and data structures that realize the functions of each unit described below. For example, the memory 304 may store software modules that realize the functions of the attack generation model learning unit 340, the attack image generation unit 350, and the AI model robustness evaluation unit 360. The constituent units of each unit and the software module may not be the same. For example, multiple units may be realized by one software module, or one unit may be realized by multiple software modules.

In one embodiment, the attack generation model learning unit 340, the attack image generation unit 350, and the AI model robustness evaluation unit 360 may be implemented in part or in whole in hardware using semiconductor devices, chips, logic gates, circuits, circuit cards, and/or other physical hardware devices instead of or in addition to a processor-based system in which a software program that realizes the functions of the attack generation model learning unit 340 is executed by a processor. In another embodiment, the attack generation model learning unit 340, the attack image generation unit 350, and the AI model robustness evaluation unit 360 may include data other than instructions or descriptions of the software program. In another embodiment, a camera, sensor, or other data input device (not shown) may be provided to directly communicate with the bus interface unit 309, the processor 302, or other hardware of the computer system 300. Details of the attack generation model learning unit 340, the attack image generation unit 350, and the AI model robustness evaluation unit 360 will be described later with reference to FIGS. 2, 3, and 4.

The computer system 300 may include a bus interface unit 309. The bus interface unit 309 provides communication between the processor 302, the memory 304, the display system 324, and an I/O bus interface unit 310. The I/O bus interface unit 310 may be connected to an I/O bus 308. Various inputs and outputs are connected to the I/O bus 308 to allow data transfer. The I/O bus interface unit 310 may communicate via the I/O bus 308 with multiple I/O interfaces (312, 314, 316, and 318), commonly known as I/O processors (IOPs) or I/O adapters (IOAs).

Display system 324 is a system that performs processing to display images on display device 326, and may include one or both of a display controller and a display memory (not shown). The display controller can provide both video and audio data to display device 326. Display system 324 may be connected to display device 326, such as a standalone display screen, a television, a tablet, or a portable device.

Computer system 300 may also include one or more sensors or other devices (not shown) configured to collect data and provide the data to processor 302. For example, computer system 300 may include biometric sensors that collect heart rate data, stress level data, etc., environmental sensors that collect humidity data, temperature data, pressure data, etc., and motion sensors that collect acceleration data, movement data, etc. Other types of sensors may also be used.

The I/O interface units (312, 314, 316, and 318) are capable of communicating with various storage or I/O devices. For example, the terminal interface 312 can be attached to user I/O devices 320, such as user output devices such as a video display device, a speaker television, and user input devices such as a keyboard, a mouse, a keypad, a touchpad, a trackball, a button, a light pen, or other pointing device. A user may use the user interface to input input data and instructions to the user I/O device 320 and the computer system 300 by operating the user input device, and to receive output data from the computer system 300. The user interface may be displayed on a display device, played through a speaker, or printed through a printer via the user I/O device 320, for example.

The storage interface 314 may be attached to one or more disk drives or directly accessed storage devices 322 (usually magnetic disk drive storage devices, but may also be an array of disk drives or other storage devices configured to appear as a single disk drive). In one embodiment, the storage device 322 may be implemented as any secondary storage device. The contents of the memory 304 may be stored in the storage device 322 and retrieved from the storage device 322 as needed. The I/O device interface 316 may provide an interface to other I/O devices such as printers, fax machines, etc. The network interface 318 may provide a communication path to allow the computer system 300 and other devices to communicate with each other. This communication path may be, for example, a network 330.

In one aspect, computer system 300 may be a device that receives requests from other computer systems (clients) without a direct user interface, such as a multi-user mainframe computer system, a single-user system, or a server computer. In other embodiments, computer system 300 may be a desktop computer, a portable computer, a laptop, a tablet computer, a pocket computer, a telephone, a smartphone, or any other suitable electronic device.

Next, the attack generation model learning unit 340 according to this embodiment will be described with reference to FIG.

Figure 2 is a block diagram of the attack generation model learning unit 340.

The attack generation model learning unit 340 has an attack encoder 103, a noise adding unit 105, a generator 107, an identifying unit 110, a loss calculation unit 111, and a parameter updating unit 112.

The original image 101 and the class condition c 102 are input to the attack encoder 103. The original image 101 is an original image in the learning dataset used to generate the AI model to be attacked. The AI model is an inference model that detects objects from images or classifies images. The original image 101 may be subjected to several processes such as image inversion, cropping, and rotation before being input to the attack encoder to perform data expansion. The class condition c 102 is information indicating the class to be attacked. Hereinafter, the class to be attacked may be referred to as the target class. The target class is, in other words, a class that the AI model erroneously detects or erroneously classifies due to the attack. The target class may be any class into which the AI model can classify an image, or may be a class indicating a background that is not classified. The attack encoder 103 is a CNN (convolutional neural network) that maps the original image 101 to a feature vector called an attack embedding e 104, and encodes the image representation of the original image 101, the class condition c 102, and an attack strategy suitable for the original image 101 into a vector. The attack encoder 103 is not particularly limited, but for example, generates an attack embedding e 104 of an attack pattern that is most suitable for the input original image 101 and class condition c 102. The most suitable attack pattern is the attack pattern that is most likely to be successful.

The noise adding unit 105 adds multiple random noises ε 106 to the attack embeddings e 104 generated by the attack encoder 103. The random noises ε may have a uniform distribution or a Gaussian distribution, but are not limited to these. The multiple attack embeddings in which different random noises ε 106 are added to the attack embeddings e 104 are input to the generator 107.

The generator 107 receives the class condition c 102 and multiple attack embeddings as input to generate multiple attack images G(e, c, ε) 108, and outputs the multiple attack images 108 to the classification unit 110. The generator 107 is a CNN that decodes the attack embeddings into the attack images 108.

The inference unit 109 is equipped with an AI model of the target of attack, and uses the AI model to detect objects from images or classify images, and obtain the detection or classification results. Hereinafter, the detection or classification results may be referred to as the inference results. The identification unit 110 identifies the attack image 108 and the original image 101, and obtains the identification results. Identification is a visual distinction made by the human eye. Hereinafter, the identification results between the attack image 108 and the original image 101 may be referred to as the identification results.

The loss calculation unit 111 calculates a loss for evaluating the attack image 108 based on the inference result by the inference unit 109 and the classification result by the classification unit 110. The parameter update unit 112 updates the parameters of the attack encoder 103, the generator 107, and the classification unit 110 so as to optimize the loss calculated by the loss calculation unit 111. Details of the calculation of the loss by the loss calculation unit 111 and the update of the parameters by the parameter update unit 112 are described below.

Although the attack image 108 is a slight modification of the original image 101, it is not preferable to make the attack image 108 completely different from the original image 101. Therefore, in order to limit the difference between the original image 101 and the attack image 108 to an appropriate range, the loss L _dif is designed as shown in formula (1). The loss L _dif (first loss) is a loss for limiting the difference between the original image 101 and the attack image 108 to an appropriate range M. Hereinafter, the loss L _dif may be referred to as the first loss.

ここで、ｘは元画像である。Ｅ_εは、元画像ｘと、複数のランダムノイズεにより生成された複数の攻撃画像Ｇ（ｅ，ｃ，ε） １０８との差分の平均値である。

Here, x is the original image, and _{E ε} is the average value of the differences between the original image x and multiple attack images G(e, c, ε) 108 generated by multiple random noises ε.

Next, the generator 107 and the classifier 110 compete against each other in a two-player minimax game, as in a normal GAN. The classifier 110 attempts to correctly classify the original image 101 and the attack image 108. Meanwhile, the generator 107 attempts to trick the classifier 110 by generating a more realistic and high-quality attack image 108. To this end, the loss L _G and the loss L _GAN are designed as shown in the following formulas (2) and (3), respectively. Hereinafter, the loss L _GAN may be referred to as the second loss.

ここで、Ｄ（ｘ）は、識別部１１０による元画像ｘが正しく識別される確率を示す。また、Ｄ（Ｇ（ｅ，ｃ，ε））は、生成された攻撃画像Ｇ（ｅ，ｃ，ε）が誤って識別される確率を示す。識別部Ｄは、Ｌ_ＧＡＮを最大化しようとする。ジェネレータ１０７は、Ｌ_Ｇを最小化しようとする。

Here, D(x) denotes the probability that the original image x is correctly classified by the classification unit 110. Also, D(G(e, c, ε)) denotes the probability that the generated attack image G(e, c, ε) is incorrectly classified. The classification unit D tries to maximize L _GAN . The generator 107 tries to minimize L _G.

Next, it can be determined whether the attack is successful or not from the inference result of object detection or image classification for the attack image 108 in the inference unit 109. For example, when the AI model is a classification task model, it can be determined that the attack is successful if the AI model erroneously classifies the attack image 108 into the target class indicated in the class condition c. Also, when the AI model is a detection task model, it can be determined that the attack is successful if the AI model erroneously detects an object of the target class from the attack image 108. Therefore, the loss L _AI is designed as shown in formula (4). Hereinafter, the loss L _AI may be referred to as the third loss.

ここで、ＡＩ（Ｇ（ｅ，ｃ，ε），ｃ）は、クラス条件ｃと、生成された攻撃画像１０８のＡＩモデルによる推論結果との差分を意味する。ＡＩモデルが分類タスクのモデルである場合、差分を最小化しようとする。ＡＩモデルが検出タスクのモデルである場合、差分を最大化しようとする。

Here, AI(G(e, c, ε), c) means the difference between the class condition c and the inference result of the AI model of the generated attack image 108. If the AI model is a model for a classification task, it tries to minimize the difference. If the AI model is a model for a detection task, it tries to maximize the difference.

Next, from the result AI(G(e,c,ε),c) of inference by the AI model for the multiple attack images G(e,c,ε), the attack image G(e,c,ε+) that is closest to the successful attack among the multiple attack images G(e,c, _ε ) is selected. For example, the attack image G(e,c,ε ₊ ) that is closest to the successful attack may be selected based on the accuracy of the inference result. Then, the loss L _e that updates the attack encoder 103 to encode a better attack embedding e is designed as shown in Equation (5). Hereinafter, the loss L _e may be referred to as the fourth loss.

ここで、円の中に点が打たれた記号は、攻撃エンベディングｅを埋め込んだ攻撃とランダムノイズεとの組み合わせ計算を意味する演算子である。これは攻撃エンベディングｅ １０４に対するランダムノイズεの追加を意味する。攻撃エンベディングｅ １０４に対するランダムノイズεの追加は、攻撃エンベディングｅ １０４に対してランダムノイズεを加算することであってもよいし、乗算することであってもよい。

Here, the symbol with a dot in a circle is an operator that means a combination calculation of an attack in which the attack embedding e is embedded and a random noise ε. This means adding the random noise ε to the attack embedding e 104. Adding the random noise ε to the attack embedding e 104 may be adding or multiplying the random noise ε to the attack embedding e 104.

The loss calculation unit 111 calculates the overall loss L shown in formula (6) from the loss functions of the first, second, third, and fourth losses described above, using hyperparameters ω ₁ , ω ₂ , and ω ₃ each representing a different weight. Hereinafter, the loss L may be referred to as the training loss.

そして、パラメータ更新部１１２は、訓練損失である損失Ｌに基づいて勾配を計算し、勾配を用いた最適化アルゴリズムによって、攻撃エンコーダ１０３、ジェネレータ１０７、および識別部１１０のＣＮＮモデルのパラメータを更新する。

Then, the parameter update unit 112 calculates a gradient based on the loss L, which is the training loss, and updates the parameters of the CNN models of the attack encoder 103, the generator 107, and the identification unit 110 by an optimization algorithm using the gradient.

When the learning of calculating the training loss and updating the parameters is repeated multiple times until a predetermined termination condition is met, the attack encoder 103 is trained to encode an appropriate attack embedding e 104 and perform a successful attack. As the termination condition, for example, a threshold value may be set for the accuracy of the inference result, or an upper limit may be set for the number of iterations. Then, the generator 107 is trained to generate a high-quality and realistic attack image 108. Also, the identification unit 110 is trained to distinguish between an actual image and the generated attack image 108. The attack encoder 103 and the trained generator 107 thus trained are used in the attack image generation unit 350. Also, the trained attack encoder 103, the trained generator 107, and the trained identification unit 110 are used in the AI model robustness evaluation unit 360.

Next, the attack image generation unit 350 will be described with reference to FIG. 3.

Figure 3 is a block diagram of the attack image generation unit 350. Referring to Figure 3, the attack image generation unit 350 has an attack encoder 103 and a generator 107.

The attack encoder 103 and the generator 107 are trained by the attack generation model learning unit 340. The original image 201 is an original image for generating the attack image 204. The class condition c 202 is information indicating a target class of the attack. The target class of the class condition c 202 may be the same as the target class indicated in the class condition c 102 used in the attack generation model learning unit 340. The attack encoder 103 can encode an optimal attack embedding e 203 for the original image 201 and the class condition c 202. The generator 107 can generate a high-quality and realistic attack image 204 based on the attack embedding e 203.

Next, the AI model robustness evaluation unit 360 will be described with reference to FIG. 4.

To evaluate an AI model, first, an attack image is generated using a trained attack encoder and a trained generator, the generated attack image is input into the AI model to be evaluated, and an inference result is obtained from the AI model. In general, the accuracy of the inference result, such as the precision rate and recall rate, is used as the evaluation criterion. However, evaluation using this evaluation method is heavily dependent on the capabilities of the generator. In other words, how well the generator is trained has a significant impact on the evaluation results. All parameters in the generator network can be obtained through training, but the number of training iterations is determined by humans based on experience. Therefore, how well the generator can be trained depends on human experience and capabilities.

In contrast, in this embodiment, the number of iterations of fine-tuning (training) the generator is used as the evaluation criterion for the robustness of the AI model. Specifically, an attack image 404 is generated using a pre-trained generator 107, and if the attack fails with that attack image 404, fine-tuning (training) of the generator 107 is repeated until an attack image 404 that is successful in the attack can be generated. The number of iterations of fine-tuning until the attack is successful is then recorded as the evaluation result 406 of the AI model. Since the more iterations of fine-tuning, the less susceptible the AI model is to attack, the number of iterations of fine-tuning is the evaluation criterion for robustness that indicates how robust the AI model is.

Figure 4 is a block diagram of the AI model robustness evaluation unit 360. Referring to Figure 4, the AI model robustness evaluation unit 360 has an attack encoder 103, a generator 107, an evaluation unit 405, an identification unit 110, a loss calculation unit 407, and a parameter update unit 408. Note that the AI model robustness evaluation unit 360 does not have a function equivalent to the noise addition unit 105 that adds random noise ε 106.

The original image 401 is the original image used to generate the attack image 404 used in the evaluation. The class condition c 402 is information indicating the target class of the attack in the evaluation. The target class of the class condition c 402 may be the same as the target class indicated in the class condition c 102 used in the attack generation model learning unit 340.

The attack encoder 103 and the identification unit 110 are trained by the attack generation model learning unit 340. The generator 107 is trained by the attack generation model learning unit 340 in the initial state.

The loss calculation unit 307 calculates the fine-tuning loss L _ft shown in formula (7) from the loss functions of the first, second, and third losses described above, using the hyperparameters ω ₁ and ω ₂ representing weights.

パラメータ更新部４０８は、微調整損失Ｌ_ｆｔに基づいて勾配を計算し、勾配を用いた最適化アルゴリズムによってジェネレータ１０７のパラメータを更新する。

The parameter update unit 408 calculates a gradient based on the fine-tuning loss L _ft , and updates the parameters of the generator 107 by an optimization algorithm using the gradient.

The evaluation unit 405 has an AI model to be evaluated, and uses the AI model to detect objects from the attack image 404 or classify the attack image 404. If the inference result indicates that the attack has failed, the loss calculation unit 407 and the parameter update unit 408 repeatedly train the generator 107 and obtain inference results from the AI model until the attack is successful. After evaluating the attack images 404 based on all original images 401, the evaluation unit 405 calculates the average accuracy of the inference results and the number of iterations, and outputs them as the evaluation result 406.

Next, the operation flow of the AI model robustness evaluation unit 360 will be explained with reference to FIG. 5.

Figure 5 is a flowchart showing the operation of the AI model robustness evaluation unit 360.

First, the necessary information including the original image 401 and class condition 402 in the evaluation dataset is input to the AI model robustness evaluation unit 360 (step S11).

Next, within the AI model robustness evaluation unit 360, the attack encoder 103 and generator 107 generate an attack embedding 403 and an attack image 404 such that the given original image 401 is misdetected or misclassified into the target class indicated in the class condition 402 (step S12).

Next, the evaluation unit 405 inputs the generated attack image 404 into the AI model and obtains an inference result by the AI model (step S13). The evaluation unit 405 then compares the inference result with the target class indicated in the class condition 402 to determine whether the attack is successful or not (step S14).

If the attack is successful, the evaluation unit 405 reflects the inference result in the evaluation result 406 (step s15). If the attack is unsuccessful, the loss calculation unit 407 calculates the fine-tuning loss L _ft based on the formula (7) (step s16), the parameter update unit 408 updates the parameters of the generator 107 based on the fine-tuning loss L _ft (step s17), increments the count of the number of iterations by +1, and returns to step s12. The AI model robustness evaluation unit 360 repeats the update process until the attack is successful, with steps s16, s17, s12, and s13 being examples of update processes, and if the attack is ultimately successful, the evaluation unit 405 reflects the inference result and the number of iterations in the evaluation result 406 (step s15).

Although the embodiments of the present invention have been described above, the present invention is not limited to the embodiments shown here, and these embodiments may be used in combination or some configurations may be changed within the scope of the technical concept of the present invention. In addition, some or all of the above embodiments include the following items. However, the present invention is not limited to the following items.

(Item 1)
The adversarial attack countermeasure support system includes an attack encoder that encodes an attack embedding from an original image, a noise adding unit that adds random noise to the attack embedding output from the attack encoder, a generator that generates an attack image for attacking an inference model using the attack embedding with the random noise added, an inference unit that performs inference on the attack image using the inference model and obtains an inference result, a discrimination unit that discriminates between the attack image and the original image and calculates the discrimination result, a loss calculation unit that calculates a training loss, which is a loss for training the attack image based on the inference result and the discrimination result, and a parameter update unit that updates the attack encoder, the generator, and the discrimination unit based on the training loss.

This allows the parameters of the attack encoder and generator to be updated based on the evaluation of attacks using attack images that use attack embeddings with added random noise, and improves the attack images, making it possible to assist in the evaluation of the robustness of AI models against attack images with a variety of attack patterns.

(Item 2)
In the adversarial attack countermeasure support system described in item 1, the loss calculation unit calculates a first loss for attempting to limit the difference between the original image and the attack image to a predetermined range, calculates a second loss for attempting to cause the classification unit to correctly identify the original image and the attack image and the generator to erroneously identify the original image and the attack image, calculates a third loss for attempting to increase the possibility that an attack against an inference by the inference model will be successful, calculates a fourth loss for attempting to reduce the distance between an attack embedding to which no random noise has been added of an attack image to which the attack is likely to be successful and an attack embedding to which random noise has been added, and calculates the training loss by summing the first loss, the second loss, the third loss, and the fourth loss with a predetermined weighting.

This makes it possible to assist in evaluating the robustness of AI models against attack images with a variety of attack patterns.

(Item 3)
The adversarial attack countermeasure support system according to item 2 further includes an attack image generation unit that generates an attack image using the attack encoder and the generator obtained by repeating the calculation of the training loss by the loss calculation unit and the update of the attack encoder, the generator, and the identification unit by the parameter update unit until a predetermined termination condition is satisfied. This makes it possible to generate an attack image that is likely to be successful in attacking an inference model.

(Item 4)
The adversarial attack countermeasure support system described in item 3 further includes a robustness evaluation unit that repeats a series of update processes, including inputting an attack image generated by the attack image generation unit to the inference model to obtain an inference result, identifying the attack image from an original image to calculate an identification result, calculating a fine-tuning loss that is a loss for fine-tuning the attack image based on the inference result and the identification result, updating the generator based on the fine-tuning loss if the attack is not successful based on the inference result, inputting an attack image generated using the updated generator to the attack image generation unit to the inference model to obtain an inference result and an identification result, until the attack is successful, and outputs the number of iterations of the update process as an evaluation result of the inference model.

This allows the generator to be repeatedly updated until the attack is successful, and the number of iterations is used as the evaluation result, making it possible to evaluate robustness in a way that is not dependent on the capabilities of the generator.

(Item 5)
In the adversarial attack countermeasure support system described in item 4, the robustness evaluation unit calculates the fine-tuning loss by summing the first loss, the second loss, and the third loss with a predetermined weighting. This makes it possible to appropriately evaluate robustness based on the number of iterations of the update process until an attack on an inference model is successful through an appropriate update process.

103...attack encoder, 105...noise addition unit, 107...generator, 109...inference unit, 110...identification unit, 111...loss calculation unit, 112...parameter update unit, 300...computer system, 302...processor, 304...memory, 306...memory bus, 308...I/O bus, 309...bus interface unit, 310...I/O bus interface unit, 312...terminal interface, 314...storage interface, 316...I/O device interface, 316...device interface, 318...network interface, 320...I/O device, 322...storage device, 324...display system, 326...display device, 330...network, 340...attack generation model learning unit, 350...attack image generation unit, 360...AI model robustness evaluation unit, 405...evaluation unit 406...evaluation result, 407...loss calculation unit, 408...parameter update unit

Claims (7)

an attack encoder that encodes an attack embedding from an original image;
a noise adding unit that adds random noise to the attack embedding output from the attack encoder;
A generator that generates an attack image for attacking an inference model using the attack embedding to which the random noise is added;
an inference unit that performs inference on the attack image using the inference model and obtains an inference result;
a classification unit that classifies the attack image and the original image and calculates a classification result;
a loss calculation unit that calculates a training loss, which is a loss for training the attack image, based on the inference result and the classification result;
a parameter updater that updates the attack encoder, the generator, and the discriminator based on the training loss;
A support system for countering hostile attacks. The loss calculation unit is
Calculating a first loss for limiting a difference between the original image and the attack image to a predetermined range;
Calculating a second loss in which the classifier tries to correctly classify the original image and the attack image and the generator tries to incorrectly classify the original image and the attack image;
Calculating a third loss to attempt to increase the likelihood of a successful attack against an inference by the inference model;
Calculating a fourth loss for reducing the distance between an attack embedding to which no random noise has been added and an attack embedding to which random noise has been added of the attack image to which the attack is likely to be successful;
calculating the training loss by summing the first loss, the second loss, the third loss, and the fourth loss with a predetermined weighting;
The hostile attack countermeasure support system according to claim 1 . and an attack image generation unit that generates an attack image using the attack encoder and the generator obtained by repeating the calculation of the training loss by the loss calculation unit and the update of the attack encoder, the generator, and the discrimination unit by the parameter update unit until a predetermined termination condition is satisfied.
The hostile attack countermeasure support system according to claim 2. a robustness evaluation unit that repeats a series of update processes, including inputting an attack image generated by the attack image generation unit into the inference model to obtain an inference result, discriminating between the attack image and an original image to calculate a discrimination result, calculating a fine-tuning loss, which is a loss for fine-tuning the attack image based on the inference result and the discrimination result, updating the generator based on the fine-tuning loss if the attack is not successful based on the inference result, inputting an attack image generated by the updated generator into the inference model to obtain an inference result and a discrimination result, until the attack is successful, and outputs the number of iterations of the update process as an evaluation result of the inference model.
The hostile attack countermeasure support system according to claim 3. the robustness evaluation unit calculates the fine-tuning loss by summing the first loss, the second loss, and the third loss with a predetermined weighting;
The hostile attack countermeasure support system according to claim 4. The computer
The attack encoder encodes the attack embedding from the original image.
adding random noise to the attack embedding output from the attack encoder;
A generator generates an attack image for attacking the inference model using the attack embedding to which the random noise has been added;
Performing inference on the attack image using the inference model to obtain an inference result;
A discrimination unit discriminates between the attack image and the original image and calculates a discrimination result;
Calculating a training loss, which is a loss for training the attack image, based on the inference result and the classification result;
updating the attack encoder, the generator, and the discriminator based on the training loss;
A method for supporting countermeasures against adversarial attacks. The attack encoder encodes the attack embedding from the original image.
adding random noise to the attack embedding output from the attack encoder;
A generator generates an attack image for attacking the inference model using the attack embedding to which the random noise has been added;
Performing inference on the attack image using the inference model to obtain an inference result;
A discrimination unit discriminates between the attack image and the original image and calculates a discrimination result;
Calculating a training loss, which is a loss for training the attack image, based on the inference result and the classification result;
updating the attack encoder, the generator, and the discriminator based on the training loss.
A program to be executed by a computer to assist in countering hostile attacks.

Images