CN118674803A - Image generation model, training method and device for image generation model - Google Patents

Abstract

The present application discloses an image generation model, a training method and a device for the image generation model, and belongs to the field of artificial intelligence technology. The image generation model includes: a conditional encoder, which is used to encode at least one original image to obtain at least one image feature information, and is used to encode the description text of each original image to obtain at least one text feature information; at least one conditional feature information is determined based on a control condition, at least one image feature information and at least one text feature information; the control condition indicates the type of the image generation task, and the conditional feature information is determined according to the type of the image generation task; a noise adding module, which is used to add noise to at least one original image based on a Gaussian noise matrix and a mask image of at least one original image to obtain at least one noise image; a diffusion model, which is used to generate at least one derivative image based on at least one conditional feature information and at least one noise image; the conditional encoder and the noise adding module are respectively connected to the diffusion model.

Description

Image generation model, image generation model training method and device

Technical Field

The present application belongs to the field of artificial intelligence technology, and specifically relates to an image generation model, and a training method and device for the image generation model.

Background Art

With the development of AIGC (Artificial Intelligence Generated Content) technology, the application of image generation tasks such as text-to-image, text and image-to-image, image-to-image, image expansion, local image modification, and image elimination has become increasingly widespread.

In the prior art, for text-generated image tasks, it is necessary to use billions of training samples to train a text-generated image model for processing; for image-generated image tasks, it is necessary to use millions of image-generated image training samples to train an image-generated image model based on the text-generated image model; or, for image expansion tasks, it is necessary to use millions of image expansion training samples to train an image expansion model based on the text-generated image model; or, for image local modification tasks, it is necessary to use millions of image local modification training samples to train an image local modification model based on the text-generated image model.

It can be seen that each time the electronic device processes an image generation task, it needs to call the corresponding model once, which is cumbersome and reduces the efficiency of the electronic device in processing image generation tasks.

Summary of the invention

The purpose of this application is to provide an image generation model, a training method and a device for the image generation model, which can improve the efficiency of electronic devices in processing image generation tasks.

In a first aspect, some embodiments of the present application provide an image generation model, including: a conditional encoder, a noise adding module and a diffusion model, wherein the conditional encoder and the noise adding module are respectively connected to the diffusion model;

A conditional encoder, used to encode at least one original image to obtain at least one image feature information, and to encode a description text of each original image to obtain at least one text feature information; and to determine at least one conditional feature information based on a control condition, at least one image feature information and at least one text feature information; the control condition is used to indicate a type of an image generation task, and the conditional feature information is determined according to the type of the image generation task;

A noise adding module, used for performing noise adding processing on at least one original image based on a Gaussian noise matrix and a mask image of at least one original image, so as to obtain at least one noisy image;

The diffusion model is used to generate at least one derivative image based on at least one conditional feature information and at least one noise image, and each derivative image corresponds to one conditional feature information and one noise image.

In a second aspect, some embodiments of the present application provide a method for training an image generation model, including:

Obtain at least two training sample groups, each training sample group including a sample image, description text of each sample image, and a mask image;

Encoding at least one sample image in at least two training sample groups to obtain at least one item of image feature information, and encoding a description text of each sample image to obtain at least one item of text feature information;

Determining at least one conditional feature information based on a control condition, at least one image feature information, and at least one text feature information; the control condition is used to indicate the type of the image generation task, and the conditional feature information is determined according to the type of the image generation task;

Based on at least one sample image, at least one mask image of the sample image and at least one conditional feature information, an initial model is trained to obtain an image generation model.

In a third aspect, some embodiments of the present application provide a training device for an image generation model, the device comprising:

An acquisition unit, used for acquiring at least two training sample groups, each training sample group including a sample image, description text of each sample image and a mask image;

an encoding unit, configured to encode at least one sample image in the at least two training sample groups acquired by the acquisition unit to obtain at least one item of image feature information, and to encode a description text of each sample image to obtain at least one item of text feature information;

a determining unit, configured to determine at least one conditional feature information based on the control condition, at least one image feature information and at least one text feature information obtained by the encoding unit; the control condition is used to indicate the type of the image generation task, and the conditional feature information is determined according to the type of the image generation task;

The training unit is used to train the initial model based on at least one sample image acquired by the acquisition unit, the mask image of at least one sample image and at least one conditional feature information determined by the determination unit to obtain an image generation model.

In a fourth aspect, some embodiments of the present application provide an electronic device comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the training method of the image generation model as described in the second aspect are implemented.

In a fifth aspect, some embodiments of the present application provide a readable storage medium storing a program or instruction, which, when executed by a processor, implements the steps of the image generation model training method as described in the second aspect.

In a sixth aspect, some embodiments of the present application provide a chip comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the training method of the image generation model as described in the second aspect.

In a seventh aspect, some embodiments of the present application provide a computer program product, which is stored in a storage medium and executed by at least one processor to implement the training method of the image generation model as described in the second aspect.

Some embodiments of the present application provide an image generation model, including a conditional encoder, a noise addition module and a diffusion model, wherein the conditional encoder and the noise addition module are respectively connected to the diffusion model; the conditional encoder is used to encode at least one original image to obtain at least one image feature information, and is used to encode the description text of each original image to obtain at least one text feature information; and based on a control condition, at least one image feature information and at least one text feature information, at least one conditional feature information is determined; the control condition is used to indicate the type of image generation task, and the conditional feature information is determined according to the type of image generation task; the noise addition module is used to perform noise processing on at least one original image based on a Gaussian noise matrix and a mask image of at least one original image to obtain at least one noise image; the diffusion model is used to generate at least one derivative image based on at least one conditional feature information and at least one noise image, each derivative image corresponding to one conditional feature information and one noise image. In the above-mentioned image generation model, conditional feature information is determined according to the type of image generation task to control the image generation model to perform different image generation tasks, and the original image is subjected to noise processing using a Gaussian noise matrix and a mask image, so that the image generation model can not only have the ability to generate images from noise, but also have the ability to predict, eliminate or modify the image content of the mask area corresponding to the mask image, so that the image generation model can adapt to different types of image generation tasks, such as text-generated images, image-generated images, text-plus-image-generated images, local image modification, image elimination, image expansion, etc. In this way, an image generation model has the ability to handle multiple types of image generation tasks, and there is no need to call multiple different models when handling different types of image generation tasks, thereby improving the efficiency of processing image generation tasks.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of the structure of an image generation model provided by some embodiments of the present application;

FIG2 is a schematic diagram of the structure of an image generation model provided by some embodiments of the present application;

FIG3 is a schematic diagram of the structure of a Transformer Block provided in some embodiments of the present application;

FIG4 is a schematic diagram of the structure of a first conditional encoder provided in some embodiments of the present application;

FIG5 is a schematic diagram of the structure of a second conditional encoder provided in some embodiments of the present application;

FIG6 is a schematic diagram of the structure of a diffuse self-attention module provided in some embodiments of the present application;

FIG7 is a schematic diagram of the structure of a diffusion model provided by some embodiments of the present application;

FIG8 is a schematic diagram of the structure of a convolution module provided in some embodiments of the present application;

FIG9 is a schematic diagram of the structure of an image encoder provided by some embodiments of the present application;

FIG10 is a schematic diagram of the structure of a deconvolution module provided in some embodiments of the present application;

FIG11 is a schematic diagram of the structure of an image decoder provided in some embodiments of the present application;

FIG12 is a flow chart of a method for training an image generation model provided in some embodiments of the present application;

FIG13 is a flowchart of a method for training an image generation model provided in some embodiments of the present application;

FIG14 is a flowchart of a method for training an image generation model provided in some embodiments of the present application;

FIG15 is a schematic diagram of a mask image provided by some embodiments of the present application;

FIG16 is a schematic diagram of two methods for determining noise addition matrices provided in some embodiments of the present application;

FIG17 is a schematic diagram of loss value calculation provided by some embodiments of the present application;

FIG18 is a schematic diagram of the structure of a training device for an image generation model provided in some embodiments of the present application;

FIG19 is a schematic diagram of the structure of an electronic device provided in some embodiments of the present application;

FIG. 20 is a schematic diagram of the hardware structure of an electronic device provided in some embodiments of the present application.

DETAILED DESCRIPTION

The following will be combined with the drawings in some embodiments of the present application to clearly describe the technical solutions in some embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

The terms "first", "second", etc. in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and/or" in the specification and claims represents at least one of the connected objects, and the character "/" generally indicates that the objects associated with each other are in an "or" relationship.

The terms "at least one (item)", "at least one of" and the like in the specification and claims of the present application refer to any one, any two or a combination of more than two of the objects included therein. For example, at least one (item) of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c" and "a, b and c", where a, b, and c can be single or multiple. Similarly, "at least two (items)" refers to two or more, and its meaning is similar to that of "at least one (item)".

First, the terms used in the specification of this application are explained.

DiT: Diffusion Transformer is a diffusion model that combines the Transformer architecture for image and video generation tasks. It can efficiently capture dependencies between data and generate high-quality results.

VAE: The full name of VAE is Variational Auto encoder, which is a generative model mainly used for unsupervised learning. VAE mainly consists of two parts: encoder and decoder. The core idea is to map the original data to the latent space through the encoder, and then restore the vector of the latent space to the original data through the decoder.

Tokenizer: A tool used to break text into words, phrases, or tokens.

Self Attention: The self-attention operator is used to calculate the attention weight of each element in the input data and recalculate the feature vector of each element based on the attention weight of each element.

Linear: A linear transformation operator used to map input data from one space to another.

Conv: Convolution operator, used to downsample the input data and reduce the size of the input data.

Deconv: Deconvolution operator, used to upsample the input data and increase the size of the input data.

Feed Forward: Feedforward operator, used to perform nonlinear transformation on the vector of each element, thereby introducing more nonlinear capabilities and improving the expressiveness of the model.

Add: Addition operator, used for addition operations.

Norm: Normalization operator, used to normalize the input data.

Multi Head Self Attention (MHSA): is a self-attention mechanism used to share information between different self-attention heads.

Segment Anything Model (SAM): A segmentation model used to segment all entities in an image.

Text-to-image task: The task of generating images based on text.

Image-to-image task: The task of generating images based on images.

Text and image generation task: the task of processing images based on text.

Local image modification task: the task of modifying the image content in a local area of an image.

Image elimination task: the task of eliminating the image content in a local area of an image.

Image expansion task: the task of completing the local area in the image.

The text processing methods provided by some embodiments of the present application are described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

The image generation model provided in some embodiments of the present application is applied to the scene of processing image generation tasks. For example, the image generation model is applied to the scene of processing image generation tasks such as text-to-image task, image-to-image task, text-to-image-to-image task, image elimination task, image local modification task or image expansion task.

For example, for the application scenario of processing text image tasks: in the related technology, the electronic device can obtain a large number of text image training samples related to the text image scenario, such as descriptive text and sample images, perform model training based on the descriptive text and sample images, obtain a model with text image function, and then call the above model with text image function to process the text image task.

For example, for the application scenario of processing image-generated-image tasks: in the related technology, when a user wants an electronic device to train a model that can realize the image-generated-image function, the electronic device first trains a text-generated-image model through the method in the above-mentioned text-generated-image scenario, and then uses the image-generated-image training samples on the basis of the text-generated-image model to train a model with the image-generated-image function, and then calls the above-mentioned model with the image-generated-image function to process the image-generated-image task.

For example, for application scenarios of processing local image modification tasks, in related technologies, when a user wants an electronic device to train a model with a local image modification function, the electronic device first trains a Vincent graph model using the method in the above-mentioned Vincent graph scenario, and then uses the image local modification training samples on the basis of the Vincent graph model to train a model with the image local modification function, and then calls the above-mentioned model with the image local modification function to process the image local modification task.

It can be seen that each time the electronic device processes an image generation task, it needs to call the corresponding model once, which is cumbersome and reduces the efficiency of the electronic device in processing image generation tasks.

Some embodiments of the present application provide an image generation model, including a conditional encoder, a noise addition module and a diffusion model, wherein the conditional encoder and the noise addition module are respectively connected to the diffusion model; the conditional encoder is used to encode at least one original image to obtain at least one image feature information, and is used to encode the description text of each original image to obtain at least one text feature information; and based on a control condition, at least one image feature information and at least one text feature information, at least one conditional feature information is determined; the control condition is used to indicate the type of image generation task, and the conditional feature information is determined according to the type of image generation task; the noise addition module is used to perform noise processing on at least one original image based on a Gaussian noise matrix and a mask image of at least one original image to obtain at least one noise image; the diffusion model is used to generate at least one derivative image based on at least one conditional feature information and at least one noise image, each derivative image corresponding to one conditional feature information and one noise image. In the above-mentioned image generation model, conditional feature information is determined according to the type of image generation task to control the image generation model to perform different image generation tasks, and the original image is subjected to noise processing using a Gaussian noise matrix and a mask image, so that the image generation model can not only have the ability to generate images from noise, but also have the ability to predict, eliminate or modify the image content of the mask area corresponding to the mask image, so that the image generation model can adapt to different types of image generation tasks, such as text-generated images, image-generated images, text-plus-image-generated images, local image modification, image elimination, image expansion, etc. In this way, an image generation model has the ability to handle multiple types of image generation tasks, and there is no need to call multiple different models when handling different types of image generation tasks, thereby improving the efficiency of processing image generation tasks.

1 is a schematic diagram of the structure of an image generation model provided by some embodiments of the present application, wherein the image generation model comprises: a conditional encoder 11, a noise adding module 12 and a diffusion model 13, wherein the conditional encoder 11 and the noise adding module 12 are respectively connected to the diffusion model 13.

In some embodiments of the present application, the conditional encoder 11 is an encoder for encoding images and texts and determining conditional feature information according to control conditions and feature information obtained by encoding.

In some embodiments of the present application, the conditional encoder 11 is used to encode at least one original image to obtain at least one image feature information, and to encode the description text of each original image to obtain at least one text feature information; and based on the control condition, at least one image feature information and at least one text feature information, at least one conditional feature information is determined; the control condition is used to indicate the type of image generation task, and the conditional feature information is determined according to the type of image generation task.

In some embodiments of the present application, each of the at least one original image may be an image including at least one entity.

Exemplarily, the original image may be an image of at least one of plants, animals, people, mountains, water, houses, computers, and the like.

In some embodiments of the present application, the description text of each original image may be used to describe the image theme of each original image.

For example, the description text of an original image may be "a man and a woman standing in front of a lawn, making a thumbs-up gesture."

In some embodiments of the present application, the description text of each original image may be used to describe the image subject of each original image and the processing operation performed on each original image.

For example, the description text of an original image may be "A man and a woman stand in front of a lawn, making a thumbs-up gesture. Delete the man in the picture."

In some embodiments of the present application, the description text of each original image may be used to describe the image subject of each original image and the image content of the mask area in each original image.

For example, the description text of an original image may be "a man and a woman standing in front of a lawn, making a thumbs-up gesture, and the picture contains a man, a woman, a balloon, a poster, and a hat". It can be understood that the mask area of the original image includes the image areas corresponding to the man, the woman, the balloon, the poster, and the hat in the original image.

In some embodiments of the present application, in combination with FIG. 1 , as shown in FIG. 2 , the conditional encoder 11 includes a first conditional encoder 111 and a second conditional encoder 112 .

It should be noted that the first conditional encoder 111 may be any image encoder capable of encoding images, and the second conditional encoder 112 may be any text encoder capable of encoding text, and some embodiments of the present application do not limit this.

In some embodiments of the present application, the first conditional encoder 111 includes an image segmentation module, a mapping layer and a self-attention module, wherein the image segmentation module and the self-attention module are respectively connected to the mapping layer.

In some embodiments of the present application, the image segmentation module is used to segment the image. The mapping layer is a model layer for mapping the input image from the original pixel space to the feature expression space. The self-attention module is a module for capturing the correlation between pixels in the image.

In some embodiments of the present application, the above-mentioned conditional encoder 11 is used to encode at least one original image to obtain at least one item of image feature information, including: the electronic device inputs at least one original image into an image segmentation module, the image segmentation module divides each original image into N original image blocks, and rearranges the pixels of each original image block into a one-dimensional vector to obtain N one-dimensional vectors of each original image, and splices the N one-dimensional vectors of each original image to obtain a pixel rearranged image of each original image input into a mapping layer; the mapping layer extracts the feature information of each pixel point in each pixel rearranged image output by the image segmentation module, and obtains at least one pixel rearranged feature vector input into a self-attention module; the self-attention module extracts features from each pixel rearranged feature vector output by the mapping layer based on the correlation between the pixels in each pixel rearranged image, and obtains the image feature information of each original image.

Wherein, N is an integer greater than 1.

In some embodiments of the present application, a one-dimensional vector is an ordered sequence, which includes each pixel in the original image.

In some embodiments of the present application, for each original image, the above-mentioned image segmentation module divides the original image into N original image blocks, and then rearranges the pixels of each original image block from left to right and from top to bottom to obtain a one-dimensional vector, that is, a new pixel arrangement, and then obtains N one-dimensional vectors, and splices the N one-dimensional vectors to obtain an N-dimensional vector, which constitutes a pixel rearranged image.

Exemplarily, the above-mentioned image segmentation module can be a patch module. Taking the resolution of the original image as [3, 512, 512] as an example, the electronic device divides each 3*32*32 pixel area in the original image into an original image block through the image segmentation module, and obtains a total of 256 original image blocks. The pixels in each original image block are then flattened in order from left to right and from top to bottom to obtain a pixel arrangement of 1*768, and finally the original image is converted into a pixel rearranged image with a pixel arrangement of 256*768.

In this way, the electronic device divides the original image with a larger resolution into original image blocks with a smaller resolution through the image segmentation module, which can reduce the subsequent calculation amount. Moreover, since the features of the pixels in the same pixel area are relatively close, the pixels of each original image block are rearranged, which can retain the feature information of the pixels in the original image block, making it easier for the image generation model to extract features from the original image.

In some embodiments of the present application, the above-mentioned mapping layer may be a Project layer, which performs a linear transformation operation.

Exemplarily, the linear transformation operation may be expressed as: y=W*x+b.

Wherein, x represents the pixel rearrangement image, y represents the pixel rearrangement feature vector, b represents the bias vector, and W represents the weight matrix. Moreover, b and W are both fixed parameters of the second encoder.

In some embodiments of the present application, for each pixel rearranged image, the electronic device performs the above-mentioned linear transformation operation on each pixel in the pixel rearranged image through a mapping layer to obtain feature information of each pixel, and combines the feature information of multiple pixels in the pixel rearranged image to obtain a pixel rearranged feature vector.

In this way, the electronic device maps the pixel-rearranged image from the original pixel space to the feature expression space through the mapping layer, that is, extracts the features of the pixel-rearranged image to obtain a pixel-rearranged feature vector, which can better represent the image content of the original image, so that the image generation model can more easily identify the entities in the original image.

In some embodiments of the present application, the association relationship between each pixel point includes the spatial dependency and semantic correlation between each pixel point. Among them, the spatial dependency refers to the relative position and distance relationship between each pixel point, and the semantic correlation refers to the content correlation between each pixel point. Among them, the content correlation refers to the correlation or correlation between different pixels in terms of content, color, brightness, etc.

For example, in the original image, adjacent pixels often have similar color and brightness values, thus forming a continuous visual perception. This continuity of color and brightness reflects the content relevance between pixels. The texture and pattern of the original image are composed of multiple pixels. If the texture and pattern between pixels remain continuous, then these pixels have a high content relevance. In some cases, the content relevance between pixels may also be reflected in the semantic information they represent. For example, in a face image, different pixels may represent different semantic units such as eyes, nose, and mouth, and the relative positions and combinations of these semantic units reflect the content relevance between pixels. Therefore, the content relevance between pixels includes the continuity of color and brightness between pixels, the continuity of texture and pattern between pixels, and the semantic relevance between pixels.

In some embodiments of the present application, the above-mentioned self-attention module may include multiple layers of Transformer Block, each layer of Transformer Block includes a self-attention operator (Self Attention), an addition operator (Add), a normalization operator (Norm) and a forward feedback operator (Feed Forward).

In some embodiments of the present application, the self-attention operator is used to calculate the attention weight of each pixel, and recalculate the feature vector of each pixel according to the attention weight of each pixel. The addition operator is used to implement addition operations, the normalization operator is used to normalize the input, and the forward feedback operator is used to perform nonlinear transformation on the input.

In some embodiments of the present application, for each pixel rearrangement feature vector, the self-attention operator converts each pixel point in the input pixel rearrangement feature vector into three vectors, namely, query vector query, key vector key and value vector value. For any pixel point in the pixel rearrangement feature vector, it is recorded as the first pixel point, and the query vector query of the first pixel point is multiplied with the key vector key of each pixel point in the pixel rearrangement feature vector to obtain the attention weight corresponding to each pixel point. The attention weight corresponding to each pixel point represents the correlation between the pixel point and the first pixel point. The attention weight corresponding to each pixel point is multiplied with the value vector value of the pixel point to obtain the vector of each pixel point, and multiple vectors are added to obtain the output vector of the first pixel point after passing through the self-attention operator. Similarly, the above operation is performed on each pixel point in the pixel rearrangement feature vector, and the output vector of each pixel point in the pixel rearrangement feature vector after being processed by the self-attention operator can be obtained. The multiple output vectors are spliced to obtain the feature vector of the pixel rearrangement feature vector after being processed by the self-attention operator, and the self-attention operator inputs the processed feature vector into the above addition operator.

In some embodiments of the present application, the addition operator adds the output of the self-attention operator to the input of the self-attention operator, that is, adds the pixel rearrangement feature vector to the feature vector output after the self-attention operator processes the pixel rearrangement feature vector, and inputs the added feature vector into the normalization operator. In this way, if the output of the previous layer retains less information, the input feature information can be retained by adding the input of the previous layer to avoid losing too much information.

In some embodiments of the present application, the above-mentioned normalization operator calculates the mean and standard deviation of each dimension according to the multidimensional eigenvalues of each pixel point in the eigenvector obtained by adding the output of the addition operator, and uses the mean and standard deviation of each dimension to normalize the eigenvalue of each pixel point in that dimension to obtain a feature matrix, and one feature matrix corresponds to one pixel rearrangement feature vector.

In some embodiments of the present application, the above-mentioned forward feedback operator includes one or more fully connected layers and an activation function layer. The forward feedback operator first performs linear calculations on the feature matrix output by the normalization operator through one or more fully connected layers, and then converts the linear output of the fully connected layer into a nonlinear output through the activation function layer, that is, performs a nonlinear transformation on the value of each pixel point to obtain the image feature information of the original image.

Exemplarily, as shown in Figure 3, it is a schematic diagram of the structure of the Transformer Block provided by some embodiments of the present application. In Figure 3, the Transformer Block includes a self-attention operator (Self Attention), an addition & normalization operator (Add&Norm) and a forward feedback operator (Feed Forward), and the self-attention operator and the forward feedback operator are respectively connected to the addition & normalization operator. The electronic device inputs the query vector query, key vector key and value vector value of each pixel point in the pixel rearrangement feature vector into the self-attention operator. After being processed by the self-attention operator, the feature vector is obtained and input into the addition & normalization operator. The addition & normalization operator performs vector addition and normalization processing on the feature vector input by the self-attention operator and the pixel rearrangement feature vector. After obtaining the feature matrix, the feature matrix is input into the forward feedback operator for nonlinear transformation. The forward feedback operator inputs the nonlinear transformation result into the addition & normalization operator. The addition & normalization operator performs vector addition and normalization processing on the input of the forward feedback operator, i.e., the feature matrix, and the output, i.e., the nonlinear transformation result, and finally outputs the image feature information of the original image.

It should be noted that the self-attention module of the first conditional encoder 111 may include a 16-layer Transformer Block, which can extract deeper and richer feature information of the original image.

Exemplarily, as shown in FIG4, it is a schematic diagram of the structure of the first conditional encoder provided by some embodiments of the present application. In FIG4, the first conditional encoder 111 includes an image segmentation module, a mapping layer and a self-attention module, and the self-attention module includes a 16-layer Transformer Block. The specific implementation of the first conditional encoder 111 encoding an original image to obtain an image feature information includes: the electronic device inputs the original image with a resolution of 512×512×3 into the image segmentation module of the first conditional encoder for image segmentation processing, and obtains a pixel rearranged image with a resolution of 256×768 and inputs it into the mapping layer. The mapping layer extracts features from the pixel rearranged image to obtain a pixel rearranged feature vector with a pixel arrangement of 256×768 and inputs it into the self-attention module. After the self-attention module processes the pixel rearranged feature vector, the image feature information (Image Embedding) of the original image with a dimension of 256×768 is obtained.

In this way, the first conditional encoder extracts features from the original image to obtain image feature information of the original image. When the image generation model processes the image generation task, it can combine the image feature information of the original image for processing, thereby improving the accuracy of processing the image generation task.

In some embodiments of the present application, the second conditional encoder 112 includes a word segmenter and a self-attention module. In addition, the word segmenter is connected to the self-attention module.

In some embodiments of the present application, the above-mentioned word segmenter is used to segment text and convert the segmentation result into a vector. The above-mentioned self-attention module is a module for capturing the association relationship between segmentations in a text.

In some embodiments of the present application, the above-mentioned conditional encoder 11 is used to encode the description text of each original image to obtain at least one item of text feature information, including: the electronic device inputs the description text of each original image into a word segmenter, the word segmenter divides the description text of each original image into at least two words, and encodes each word to obtain a text feature vector of each description text and inputs it into a self-attention module; the self-attention module extracts features from the text feature vector of the description text obtained by the word segmenter based on the association relationship between the words in the description text, and obtains text feature information of the description text.

Exemplarily, the tokenizer first divides the description text into meaningful tokens such as words or characters according to the tokenization rules. In addition, the tokenization rules may be different for different languages and tasks. For example, in English, tokens can usually be segmented by spaces or punctuation marks; in Chinese, since there are no obvious separators between words, a tokenization algorithm or a dictionary is required for tokenization. After the description text is segmented into multiple tokens, each token can be mapped to a unique integer ID, and the description text is converted into an integer sequence, that is, the text feature vector of the description text is obtained.

Exemplarily, the word segmenter in the second conditional encoder 112 may be a Tokenizer, which segments the description text into multiple words, and maps the vector corresponding to each word to obtain a text feature vector with a dimension of [128, 768]. 128 represents the maximum number of words supported by the text encoder, and 768 is the length of the vector corresponding to each word.

Furthermore, after the description text is segmented, stop words in the segmentation can be removed, and then the vector corresponding to each segmentation can be mapped, which can reduce the amount of data that the model needs to process and improve processing efficiency.

In some embodiments of the present application, the above-mentioned self-attention module may include multiple layers of Transformer Block, each layer of Transformer Block includes a self-attention operator (Self Attention), an addition operator (Add), a normalization operator (Norm) and a forward feedback operator (Feed Forward).

It should be noted that the structure of the self-attention module of the second conditional encoder 112 is the same as the structure of the self-attention module of the first conditional encoder 111 (as shown in FIG3 ), and the processing process of the input text feature vector by the self-attention module of the second conditional encoder 112 is also the same as the processing process of the input pixel rearrangement feature vector by the self-attention module of the first conditional encoder 111. The specific implementation can be found in the above-mentioned relevant description, and the present application will not repeat it here.

It should be noted that the self-attention module of the second conditional encoder 112 may include a 12-layer Transformer Block, which can extract deeper and richer feature information describing the text.

Exemplarily, as shown in FIG5 , it is a schematic diagram of the structure of the second conditional encoder provided by some embodiments of the present application. In FIG5 , the second conditional encoder 112 includes a word segmenter and a self-attention module, and the self-attention module includes a 12-layer Transformer Block. The specific implementation of the second conditional encoder 112 encoding the description text of an original image to obtain a text feature information includes: the electronic device inputs the description text "The sun is shining, and there is a puppy running on the grass" into the word segmenter of the second conditional encoder, and the word segmenter processes the description text to obtain a text feature vector with a dimension of 128×768 and inputs it into the self-attention module. After the self-attention module processes the text feature vector, it obtains text feature information (Text Embedding) with a dimension of 128×768 of the description text.

In this way, the second conditional encoder extracts features from the description text to obtain text feature information of the description text. When the image generation model processes the image generation task, it can combine the text feature information of the description text to generate a derivative image that conforms to the description of the description text or process the original image based on the description text, thereby improving the accuracy of processing the image generation task.

In some embodiments of the present application, as shown in FIG2 , the second conditional encoder 121 includes a text encoder small and a text encoder big, both of which include a convolution module and a self-attention module, and the convolution modules of the two are the same, and the self-attention modules are composed of Transformer Blocks. Among them, the self-attention module of the text encoder small includes 12 layers of Transformer Blocks, and the self-attention module of the text encoder big includes 36 layers of Transformer Blocks.

Exemplarily, when the image generation model includes two second conditional encoders, a text encoder small and a text encoder big, the text encoder small encodes the description text to obtain text feature information 1, and the text encoder big encodes the description text to obtain text feature information 2, that is, the second conditional encoder 112 can encode two types of text feature information of the description text.

It should be noted that the text encoder small is a relatively lightweight encoder that still has feature representation capabilities, and is suitable for electronic devices with limited computing power, such as mobile communication devices, car equipment, etc. The text encoder big is an encoder with more parameters, stronger modeling and expression capabilities, and can handle more complex description texts. It is suitable for high-performance electronic devices, such as cloud graphics processing units (GPUs), servers, etc.

In some embodiments of the present application, the control condition is used to indicate the type of the image generation task, and the control conditions corresponding to different types of image generation tasks may be the same or different.

Exemplarily, for the task of text-to-image generation, the control condition is not to change the text feature information and to set the image feature information to 0. For image processing tasks such as image-to-image generation, local image modification, etc., the control condition is not to change the text feature information and image feature information.

It should be noted that the control conditions are used in the model training process of training the image generation model, and no control conditions are used when using the trained image generation model to process image generation tasks.

Exemplarily, during the model training process of training the image generation model, the electronic device may use the control condition according to a preset probability. For example, during the model training process, there is a 46.5% probability of using the control condition indicating the Vincent figure task, and a 53.5% probability of using the control condition indicating the image processing task.

In some embodiments of the present application, the conditional encoder 11 is specifically used to: when the type of the image generation task is a text-to-image task, determine at least one item of text feature information as at least one item of conditional feature information; when the type of the image generation task is an image processing task, determine at least one item of image feature information and at least one item of text feature information as at least one item of conditional feature information; wherein the image processing task includes at least one of the following: an image-to-image task, a text-to-image task, an image elimination task, an image local modification task, and an image expansion task.

Exemplarily, when the type of the image generation task is a text image task, the control condition is not to change the text feature information and to set the image feature information to 0, then the conditional encoder 11 determines at least one item of text feature information as at least one item of conditional feature information. When the type of the image generation task is an image processing task, the control condition is not to change the text feature information and the image feature information, then the conditional encoder 11 determines at least one item of text feature information and at least one item of image feature information as at least one item of conditional feature information.

Exemplarily, for an original image and a description text of the original image, when the type of the image generation task is a text image task, text feature information of the description text of the original image is determined as the conditional feature information.

Exemplarily, for an original image and a description text of the original image, when the type of the image generation task is an image modification task, image feature information of the original image and text feature information of the description text of the original image are determined as conditional feature information.

In this way, by using control conditions, the image generation model can determine the conditional feature information and perform subsequent processing according to the type of image generation task, so that an image generation model can have the ability to handle multiple types of image generation tasks, thereby improving the processing efficiency of image generation tasks.

Further, for an original image and a description text of the original image, in the case where two types of text feature information of the description text are encoded by the second conditional encoder 112. For the text image task, the control condition is not to change the two types of text feature information and set the image feature information to 0, or the control condition is not to change any type of text feature information and set the other type of text feature information and the image feature information to 0. For the image processing task, the control condition is not to change the image feature information and the two types of text feature information, or the control condition is not to change the image feature information and any type of text feature information and set the other type of text information to 0.

In some embodiments of the present application, when the type of the image generation task is a text image task, the conditional encoder 11 may determine any one of the two text feature information of the description text of the original image as the conditional feature information; or, the conditional encoder 11 may determine the two text feature information of the description text of the original image as the conditional feature information. When the type of the image generation task is an image processing task, the conditional encoder 11 may determine the image feature information of the original image and any one of the text feature information of the description text of the original image as the conditional feature information; or, the conditional encoder 11 may determine the image feature information of the original image and the two text feature information of the description text of the original image as the conditional feature information.

In some embodiments of the present application, the noise adding module 12 of the above-mentioned image generation model is used to perform noise adding processing on at least one original image based on a Gaussian noise matrix and a mask image of at least one original image to obtain at least one noisy image.

In some embodiments of the present application, the mask image of the original image is used to represent the mask region in the original image. It is understandable that after the mask image of the original image is acquired, the mask region in the original image can be determined.

Exemplarily, the mask image can be an array or a matrix, the dimension of the array or the matrix is the same as the dimension of the original image, that is, the number of element points in the array or the matrix is the same as the number of pixels in the original image, and the arrangement of the element points in the array or the matrix is the same as the arrangement of the pixels in the original image, that is, in the array or the matrix, each element point corresponds to a pixel point in the original image. And the value of each element point indicates whether the pixel point corresponding to the element point is a masked pixel point. For example, the electronic device can set the element point with a value of 1 to correspond to the pixel point of the mask area, which needs to be processed in the subsequent process of this application, and the element point with a value of 0 corresponds to the pixel point of the unmasked area, which does not need to be processed in the subsequent process of this application.

Exemplarily, the electronic device may also use a binary or Boolean image of the same size as the original image to represent the mask image. In addition, in the binary or Boolean image, the black portion represents the masked area, and the white portion represents the unmasked area; or the white portion represents the masked area, and the black portion represents the unmasked area.

For example, assuming that the resolution of the original image is [3, 512, 512], it means that the original image includes 512×512 pixels, and the value of each pixel is a 3D vector, and the 3D vector represents the RGB value of the pixel. For example, if the mask image is a binary image, and the black part represents the masked area, and the white part represents the unmasked area, then the mask image can also be an image with a resolution of [3, 512, 512], indicating that the mask image includes 512×512 pixels, and the value of each pixel is a 3D vector, the 3D vector of each pixel in the masked area of the image is the RGB value corresponding to black, and the 3D vector of each pixel in the unmasked area is the RGB value corresponding to white.

In some embodiments of the present application, the Gaussian noise matrix is a noise matrix that obeys a normal distribution and can be obtained by random sampling of an electronic device.

In some embodiments of the present application, the denoising module 12 is used to perform denoising processing on at least one original image based on a Gaussian noise matrix and a mask image of at least one original image to obtain at least one noisy image, including: the denoising module 12 performs weighted summation of the Gaussian noise matrix and the mask image of at least one original image to obtain at least one denoised matrix, and performs weighted summation of at least one denoised matrix and at least one original image to obtain at least one noisy image.

In some embodiments of the present application, each noisy image corresponds to a noise matrix and an original image.

In some embodiments of the present application, the noise adding module 12 may determine the noise adding matrix by the following formula (1):

z'＝Weight*GNM+(1-Weight)*Mask (1)

In formula (1), z' represents the noise matrix, Weight represents the weight, GNM represents the Gaussian noise matrix, and Mask represents the mask image.

It should be noted that Weight is obtained by adjusting the model parameters through back propagation during model training.

Exemplarily, the weight can take any value from [0 to 1], including 0 and 1. If the weight is 1, it can be understood as the original image is subjected to noise processing by the Gaussian noise matrix; if the weight is 0, it can be understood as the original image is subjected to noise processing by the mask image; if the weight is greater than 0 and less than 1, it can be understood as the original image is subjected to noise processing by combining the Gaussian noise matrix and the mask image.

It should be noted that the value of each element in the Gaussian noise matrix can be a one-dimensional or multi-dimensional vector, and the value of each element in the mask image can also be a one-dimensional or multi-dimensional vector. When the noise adding module 12 weights and sums the Gaussian noise matrix and the mask image, if the dimension of the Gaussian noise matrix is the same as the dimension of the mask image, that is, the number of element points included in the Gaussian noise matrix is the same as the number of element points included in the mask image, and the arrangement of the element points included in the Gaussian noise matrix is the same as the arrangement of the element points included in the mask image, and the length of the value of each element point in the Gaussian noise matrix is the same as the length of the value of each element point in the mask image, then the noise adding module 12 directly inputs the Gaussian noise matrix and the mask image into the above formula (1) to determine the noise adding matrix. However, if the dimension of the Gaussian noise matrix is different from the dimension of the mask image, the noise adding module 12 can perform dimensionality reduction processing on the one with higher dimension to make the two dimensions the same, and then input the Gaussian noise matrix and the mask image with the same dimension into the above formula (1) to determine the noise adding matrix.

For example, when the dimension of the Gaussian noise matrix is [16, 64, 64], assuming that the dimension of the mask image is [3, 512, 512], the denoising module 12 can perform dimensionality reduction processing on the mask image to obtain a mask image with a dimension of [16, 64, 64].

In some embodiments of the present application, the noise adding module 12 may determine the noise image by the following formula (2):

In formula (2), NoiseLatent represents the noise image, _x0 represents the original image, and z' represents the noise matrix. is the nonlinear coefficient and is a fixed value.

It should be noted that the value of each element in the noise matrix can be a one-dimensional or multi-dimensional vector, and the value of each pixel in the original image can also be a one-dimensional or multi-dimensional vector. When the noise module 12 weights and sums the noise matrix and the original image, if the dimension of the noise matrix is the same as the dimension of the original image, that is, the number of element points included in the noise matrix is the same as the number of pixel points included in the original image, and the arrangement of the element points included in the noise matrix is the same as the arrangement of the pixel points included in the original image, and the length of the value of each element point in the noise matrix is the same as the length of the value of each pixel point in the original image, then the noise module 12 directly inputs the noise matrix and the original image into the above formula (2) to determine the noise image. If the dimension of the noise matrix is different from the dimension of the original image, the higher-dimensional side can be subjected to dimensionality reduction processing to make the two dimensions the same, and then the noise matrix and the original image with the same dimension are input into the above formula (2) to determine the noise image.

For example, when the dimension of the noise matrix is [16, 64, 64], assuming that the dimension of the original image is [3, 512, 512], the noise adding module 12 can perform dimensionality reduction processing on the original image to obtain an original image with a dimension of [16, 64, 64].

In this way, the denoising module 12 adds noise to the original image by using only a Gaussian noise matrix, only a mask image, or a combination of a Gaussian noise matrix and a mask image. There are many forms of the denoising matrix, and there are also many forms of the noisy image obtained by denoising the original image. Therefore, the image generation model can learn the ability to reconstruct images from multiple noises, thereby improving the performance of the model.

It should be noted that the larger the dimension (that is, the more pixels or the longer the length of the value of each pixel), the greater the amount of calculation. Since the dimension of the original image is usually large, and the dimension of the Gaussian noise matrix may be small, the original image can be feature compressed first, and then the original image after feature compression can be denoised according to the Gaussian noise matrix and the mask image. This can reduce the amount of calculation for the denoising module 12 to perform the denoising process, reduce resource usage, and improve processing efficiency.

In some embodiments of the present application, the diffusion model 13 of the above-mentioned image generation model is used to generate at least one derivative image based on at least one conditional feature information and at least one noise image, and each derivative image corresponds to one conditional feature information and one noise image.

Exemplarily, the diffusion model 13 may be a Diffusion Model.

In some embodiments of the present application, the diffusion model 13 is used to generate a derivative image based on a conditional feature information and a noise image, including: the diffusion model 13 obtains the feature information of at least one noise image, and fuses the at least one conditional feature information and the feature information of at least one noise image to obtain at least one fused feature information, each fused feature information corresponds to a conditional feature information and feature information of a noise image; the diffusion model 13 determines the noise matrix of at least one noise image based on the at least one fused feature information and at least one noise vector, each noise vector represents the noise level of a noise image; the diffusion model determines the derivative image based on the noise matrix and the noise image.

In some embodiments of the present application, the feature information of the at least one noise image may be represented in the form of a feature matrix.

In some embodiments of the present application, the above-mentioned diffusion model 13 may include an image segmentation module, a linear transformation operator, a splicing operator, a diffusion self-attention module, a layer normalization operator and a convolution operator.

In some embodiments of the present application, the diffusion model 13 obtains feature information of at least one noise image, and fuses at least one conditional feature information and feature information of at least one noise image to obtain at least one fused feature information, including: the noise adding module 12 inputs at least one noise image into the image segmentation module, and the image segmentation module divides each noise image into M noise image blocks and inputs them into the linear transformation operator; the linear transformation operator extracts the feature information of each noise image block processed by the image segmentation module, and adds a position coding vector to obtain feature information of at least one noise image and inputs it into the splicing operator, and the linear transformation operator performs linear transformation processing on at least one conditional feature information to obtain at least one conditional feature information after linear transformation, and the splicing operator splices the feature information of at least one noise image processed by the linear transformation operator with at least one conditional feature information after linear transformation to obtain fused feature information.

Wherein, M is an integer greater than 1, and M is set according to actual needs.

Exemplarily, the image segmentation module may be a patching module, the linear transformation operator may be linear, and the concatenation operator may be concat.

In some embodiments of the present application, for each noise image, the noise addition module 12 inputs the noise image into an image segmentation module, the image segmentation module divides the noise image into M noise image blocks and then inputs the blocks into a linear transformation operator, the linear transformation operator extracts the feature information of each noise image block processed by the image segmentation module, and splices the feature information of the M noise image blocks according to the position of each noise image block in the noise image to obtain the feature information of the noise image.

Furthermore, the above-mentioned image segmentation module can also add a position encoding vector in the process of splicing the feature information of the M noise image blocks to mark the position sequence of each noise image block to obtain the feature information of the noise image.

For example, taking the noise image as a feature matrix with a dimension of [16, 64, 64] as an example, the image segmentation module divides the noise image into multiple noise image blocks, and then extracts the feature information of each noise image block through the above-mentioned linear transformation operator, and adds a position encoding vector (Position Embedding) to obtain the feature information of the noise image with a dimension of 4096*768.

In some embodiments of the present application, the conditional encoder 11 inputs the conditional feature information into a linear transformation operator. When the conditional feature information includes one item of text feature information, the linear transformation operator performs a linear transformation on the one item of text feature information to obtain the conditional feature information after linear transformation.

In some embodiments of the present application, when the conditional feature information includes text feature information and image feature information, the linear transformation operator converts the image feature information and the text feature information into two feature matrices of the same dimension, and then adds the features at the same position in the two feature matrices to obtain the conditional feature information after linear transformation.

In some embodiments of the present application, the linear transformation operator inputs the conditional feature information after the linear transformation and the feature information of the noise image into the above-mentioned splicing module, and the splicing model splices the conditional feature information after the linear transformation with the feature information of the noise image in dimension to obtain fused feature information (Input Tokens).

Exemplarily, taking the case where the conditional feature information includes image feature information and text feature information, and the image feature information is a feature matrix with a dimension of 256*768, the text feature information is a feature matrix with a dimension of 128*768, and the feature information of the noise image is a feature matrix with a dimension of 4096*768, the image feature information and the text feature information are converted into a feature matrix with a dimension of 410*768 through a linear transformation operator, and then the two feature matrices are added to obtain the conditional feature information after the linear transformation with a dimension of 410*768, and then the conditional feature information after the linear transformation with a dimension of 410*768 is spliced with the feature information of the noise image with a dimension of 4096*768 through a splicing operator in terms of dimension to obtain a feature matrix with a dimension of 4506*768, i.e., the fused feature information.

In some embodiments of the present application, when the above-mentioned diffusion model 13 determines the noise matrix of at least one noisy image based on at least one fused feature information and at least one noise vector, it includes: the splicing operator inputs the fused feature information into the diffusion self-attention module, the diffusion self-attention module performs feature extraction on the noise vector to obtain a set of scaling factors, and performs feature scaling on the fused feature vector based on the set of scaling factors to obtain the scaled feature information input into the layer normalization operator, the layer normalization operator performs normalization processing on the scaled feature information to obtain the noise feature information input into the convolution operator, the convolution operator performs channel reduction processing on the noise feature information to obtain the noise matrix of the noise image.

In some embodiments of the present application, the noise adding vector may be a set of preset constant vectors.

Exemplarily, the noise vector may be 0123456789…999.

Exemplarily, the above-mentioned diffuse self-attention module can be a DiT Block or a Large DiT Block.

Exemplarily, as shown in FIG6 , it is a schematic diagram of the structure of the diffusion self-attention module provided by some embodiments of the present application. In FIG6 , the diffusion self-attention module includes a multilayer perceptron (Multilayer Perceptron, MLP), a layer normalization operator (Layer Norm), a multi-head self-attention mechanism (Multi Head Self Attention, MHSA) layer, a forward feedback operator, a feature scaling layer (scale) and a residual calculation layer (shift). After the electronic device inputs the noise vector (Timestep) and the fused feature information (Attention Embedding) into the diffusion self-attention module, the processing process of the diffusion self-attention module is as follows:

a. The multi-layer perceptron extracts features from the noise vector and obtains six scaling factors, namely α1, α2, β1, β2, γ1 and γ2. The concatenation operator inputs the fused feature information into the normalization operator layer for normalization, and then the feature scaling & residual calculation layer performs feature scaling on the normalized fused feature information using scaling factors β1 and γ1. The feature scaling calculation process is Attention Embedding*γ1+β1;

b. Input the output of step a into the multi-head self-attention mechanism layer for self-attention calculation to capture the correlation between the conditional feature information and the feature information of the noise image;

c. Input the output of step b into the feature scaling layer by scaling factor α1, and then perform residual calculation with the fused feature information;

d. Normalize the output input layer normalization operator of step c, and then perform feature scaling through the feature scaling & residual calculation layer using scaling factors β2 and γ2;

e. Input the output of step d into the forward feedback operator, perform linear calculation on the input through one or more fully connected layers, and then perform nonlinear transformation on the linear output of the fully connected layer through the activation function layer to obtain a nonlinear output to increase the nonlinear representation capability of the feature;

f. Input the output of step e into the feature scaling layer and perform feature scaling by the scaling factor α2, and perform residual calculation with the output of step c to obtain the scaled feature information as the output of the diffuse self-attention module.

It should be noted that in some embodiments of the present application, the above-mentioned diffusion model may include a 16-layer diffusion self-attention module, which is used to deepen the network and increase the number of parameters of the model, so that the model has more parameters to learn more knowledge and increase the fitting ability of the model.

In some embodiments of the present application, the diffuse self-attention module inputs the scaled feature information into a layer normalization operator, and the layer normalization operator first performs a dimensionality transformation on the scaled feature information, and then normalizes the result of the dimensionality transformation to obtain noise feature information.

Exemplarily, the layer normalization operator of the diffusion model 13 may be Layer Norm.

Exemplarily, the dimensionality transformation process can be understood as a reshape operation, and the normalization process can be understood as a Layer Norm operation.

For example, assuming that the scaled feature information is a feature matrix with a dimension of 4506*768, after the above-mentioned layer normalization operator performs reshape operation and Layer Norm operation, a feature matrix with a dimension of [768, 64, 64] is obtained, that is, the noise feature information.

In some embodiments of the present application, the layer normalization operator inputs the noise feature information into the convolution operator, and the convolution operator reduces the number of channels of each element in the noise feature information without changing the size of the noise feature information, thereby obtaining a feature matrix with fewer channels, which is called the noise matrix of the noise image (Predict Latent).

Exemplarily, the above-mentioned convolution operator may be Conv_out.

Exemplarily, assuming that the noise feature information is a feature matrix with a dimension of [768, 64, 64], after the convolution operator reduces the number of channels of the feature matrix, a noise matrix with a dimension of [32, 64, 64] is obtained.

In this way, in the diffuse self-attention module, the noise vector is used to continuously add noise to the noisy image until a noise matrix that is completely noise is obtained, that is, the noise in the noisy image is predicted.

In some embodiments of the present application, the process of obtaining the noise matrix from the noise image by the diffusion model is a diffusion process, and after the convolution operator outputs the noise matrix, the noise matrix can be processed according to the inverse diffusion process to obtain a derivative image, wherein the inverse diffusion process is in the opposite direction to the diffusion process.

Exemplarily, after the convolution operator obtains the noise matrix, the noise matrix is input into the convolution operator, layer normalization operator and diffusion self-attention module of the diffusion model in sequence in the direction opposite to the diffusion process, and finally a derived image is output.

Exemplarily, as shown in Figure 7, it is a structural diagram of the diffusion model provided by some embodiments of the present application. In Figure 7, the diffusion model 13 includes an image segmentation (patching) module, a linear transformation operator (linear), a concatenation operator (concat), a diffusion self-attention module (Large DiT Block), a layer normalization operator (Layer Norm) and a convolution operator (Conv_out).

Next, the process of generating a noise matrix by the diffusion model 13 is explained in conjunction with Figure 7: Taking the conditional feature information including image feature information and text feature information as an example, the conditional encoder 11 inputs the conditional feature information with a dimension of 410*768 into the linear transformation operator, and the linear transformation operator performs linear transformation and feature combination on the image feature information and the text feature information to obtain the conditional feature information (Condition Embedding) after linear transformation, with a dimension of 410*768; the noise image (Noise Latent) with a resolution of 16*64*64 is input into the image segmentation module to be segmented into M image blocks, and the M image blocks are input into the linear transformation operator for feature extraction to obtain the feature information of each image block, and then the position encoding vector (PositionEmbedding) is added to obtain the feature information of the noise image, with a dimension of 4096*768. The conditional feature information (Condition Embedding) after linear transformation and the feature information of the noise image are input into the splicing operator for splicing to obtain the fused feature information (Attention Embedding) with a dimension of 4506*768. The fused feature information and the noise vector (Timestep) with a dimension of 1*768 are input into the diffusion self-attention module including 16 layers of Large DiT Block for processing to obtain scaled feature information with a dimension of 4506*768. The scaled feature information is input into the layer normalization operator for reshape operation and Layer Norm operation to obtain the noise feature information with a dimension of [768, 64, 64]. The noise feature information is input into the convolution operator to reduce the number of channels to obtain the noise matrix (Predict Noise) of the noise image with a dimension of [32, 64, 64].

In this way, the image generation model includes a conditional encoder, a noise addition module and a diffusion model. The conditional encoder determines the conditional feature information according to the control condition, text feature information and image feature information to control the image generation model to perform different types of image generation tasks, and in the noise addition module, the original image is subjected to noise processing using a Gaussian noise matrix and a mask image to obtain a noise image input to the diffusion model to generate a derivative image, so that the image generation model can not only have the ability to generate images from noise, but also have the ability to predict, eliminate or modify the image content of the mask area corresponding to the mask image, so that the image generation model can adapt to different types of image generation tasks, that is, an image generation model has the ability to handle multiple types of image generation tasks, and there is no need to call multiple different models when handling different types of image generation tasks, thereby improving the efficiency of processing image generation tasks.

In some embodiments of the present application, as shown in FIG. 2 , the image generation model further includes an image encoder 14 and an image decoder 15 , the image encoder 14 is connected to the noise addition module 12 , and the image decoder 15 is connected to the diffusion model 13 .

In some embodiments of the present application, the image encoder 14 is used to perform feature compression on at least one original image to obtain at least one image feature matrix.

In some embodiments of the present application, each image feature matrix in the at least one image feature matrix corresponds to an original image, and the number of pixels in each image feature matrix is less than the number of pixels in the original image.

Exemplarily, taking the dimension of the original image as [3, 512, 512], the image encoder 14 performs feature compression on the original image to obtain an image feature matrix with a dimension of [16, 64, 64].

In some embodiments of the present application, the image encoder 14 may be any encoder that can implement a feature compression function, that is, map a high-dimensional input to a low-dimensional latent variable.

Exemplarily, the image encoder 14 may be a VAE encoder (VAE-Encoder), and feature compression of the original image may be understood as compressing the original image from a pixel space to a latent space.

In some embodiments of the present application, the above-mentioned image encoder 14 may include a convolution module and a self-attention module, and the convolution module is connected to the self-attention module.

In some embodiments of the present application, the convolution module of the above-mentioned image encoder 14 is used to perform feature compression on the original image, and the self-attention module of the above-mentioned image encoder 14 is used to model element points at different positions within the feature without changing the feature dimension, so as to capture the dependency between the element points, so that the obtained image feature matrix can include more information of the original image and better characterize the original image.

In some embodiments of the present application, the image encoder 14 is used to perform feature compression on at least one original image to obtain at least one image feature matrix, including: the electronic device inputs at least one original image into the convolution module of the image encoder 14, the convolution module performs preliminary feature compression on at least one original image to obtain at least one compressed image feature matrix, the number of pixels in each compressed image feature matrix is greater than the number of pixels in each image feature matrix; the convolution module inputs at least one compressed image feature matrix into the self-attention module in the image encoder, the self-attention module performs feature extraction on each compressed image feature matrix based on the correlation between the pixels in each compressed image feature matrix to obtain at least one image feature matrix.

In some embodiments of the present application, the convolution module may include a convolution layer and a nonlinear function activation layer. For an original image, the convolution layer is used to perform preliminary feature compression on the original image, reduce the dimension of the original image, and obtain a compressed image feature matrix, and the nonlinear function activation layer is used to perform nonlinear transformation on each pixel in the compressed image feature matrix to obtain an image feature matrix.

Exemplarily, the convolution module may be ConvBlock, the convolution layer may be Conv, and the nonlinear function activation layer may be Relu.

It should be noted that the size, step size and number of output channels of the convolution kernel in the above convolution layer are predefined. In addition, the convolution layer may include multiple convolution kernels, and the number of convolution kernels is the same as the number of output channels. The size of the convolution kernel includes the height and width of the convolution kernel.

In some embodiments of the present application, after the original image is input into the convolution layer of the above-mentioned convolution module, the convolution kernel in the convolution layer slides on the original image according to the step size. Each time it slides, the convolution kernel and the image block it covers are dot-producted (i.e., the values of the element points at the corresponding positions are multiplied and then summed). Then, a value can be obtained each time it slides, which is the convolution result of the current position and the convolution kernel. After the convolution kernel slides across the entire original image, a two-dimensional output feature map can be obtained, that is, a two-dimensional output feature map will be obtained for each convolution kernel. After splicing the output feature maps of multiple convolution kernels, a compressed image feature matrix can be obtained. The dimension of the compressed image feature matrix is [number of convolution kernels, new height, new width].

It should be noted that the number of convolution kernels in the above convolution layer is the number of output channels of the convolution layer, and the above new height and new width depend on the size of the original image, the size of the convolution kernel and the step size. Exemplarily, the new height and new width can be calculated by the following formula: new height = (height of the original image - height of the convolution kernel) + 2/step size + 1; new width = (width of the sample image - width of the convolution kernel) + 2/step size + 1.

In some embodiments of the present application, the convolution layer inputs the compressed image feature matrix into the nonlinear function activation layer, and the nonlinear function activation layer performs a nonlinear transformation on the value of each pixel in the compressed image feature matrix. Specifically, for each pixel in the compressed image feature matrix, if the value of the pixel is less than 0, the value of the pixel is replaced with 0, and if the value of the pixel is greater than 0, the value of the pixel remains unchanged, and finally the image feature matrix is output.

In some embodiments of the present application, the above-mentioned convolution module may include multiple convolution layers and multiple nonlinear function activation layers, and each convolution layer is connected to a nonlinear function activation layer, that is, the number of convolution layers is the same as the number of nonlinear function activation layers.

Exemplarily, as shown in Figure 8, it is a structural diagram of a convolution module provided in some embodiments of the present application. In Figure 8, the convolution module includes three convolution layers and three nonlinear function activation layers. The three convolution layers are convolution layer 1_1 (Conv1_1), convolution layer 1_2 (Conv1_2) and convolution layer 1_3 (Conv1_3), and the three nonlinear function activation layers are nonlinear function activation layer 1_1 (Relu1_1), nonlinear function activation layer 1_2 (Relu1_2) and nonlinear function activation layer 1_3 (Relu1_3). The convolution kernel size of convolution layer 1_1 is 3x3, the stride is 2, the number of output channels is 4, and the number of convolution kernels is also 4. It is then connected to a nonlinear function activation layer 1_1. The nonlinear function activation layer 1_1 is then connected to a convolution layer 1_2. The convolution kernel size of convolution layer 1_2 is 3x3, the stride is 1, the number of output channels is 4, and the number of convolution kernels is also 4. It is then connected to a nonlinear function activation layer 1_2. The nonlinear function activation layer 1_2 is then connected to a convolution layer 1_3. The convolution kernel size of convolution layer 1_3 is 3x3, the stride is 1, the number of output channels is 4, and the number of convolution kernels is also 4. It is then connected to a nonlinear function activation layer 1_3.

It should be noted that the above convolution layer 1_1 is mainly used to extract low-level features in the original image, such as edges, colors, textures, etc. The above convolution layer 1_2 is mainly used to further combine features based on the low-level features extracted by the previous layer to extract more complex image structures, such as textures or simple shapes, etc. The above convolution layer 1_3 is mainly used to extract higher-level semantic information, such as entity categories, scenes, entity contents, etc., based on the features extracted by the previous two layers.

Exemplarily, in FIG8 , taking the dimension of the original image as [3, 512, 512] as an example, the original image with the dimension of [3, 512, 512] is input into the convolution layer 1_1 for feature extraction, and the edges and line directions of the original image can be extracted, and a feature matrix with the dimension of [4, 256, 256] is obtained and output, and then the feature matrix is input into the nonlinear function activation layer 1_1 to perform a nonlinear transformation on the value of each pixel point, and a feature matrix with the dimension of [4, 256, 256] is output, and then the feature matrix is input into the convolution layer 1_2 for further feature extraction, and larger texture blocks or simple shapes such as spots in the original image can be extracted. , stripes, corners, etc., to obtain a feature matrix with a dimension of [4, 256, 256] and output it, then input the feature matrix into the nonlinear function activation layer 1_2 to perform a nonlinear transformation on the value of each pixel, and output a feature matrix with a dimension of [4, 256, 256]. Then input the feature matrix into the convolution layer 1_3 for further feature extraction, and the specific shape and category of the entity in the original image can be extracted, and a feature matrix with a dimension of [4, 256, 256] is obtained and output, then input the feature matrix into the nonlinear function activation layer 1_3 to perform a nonlinear transformation on the value of each pixel, and output a feature matrix with a dimension of [4, 256, 256].

In some embodiments of the present application, the self-attention module of the above-mentioned image encoder 14 includes 4 layers of TansformerBlock. In this way, different levels of association relationships between pixels in the compressed image feature matrix can be extracted at each layer, so that the extraction of association relationships is richer, and an image feature matrix with richer feature information is obtained.

It should be noted that, for the structural diagram of Tansformer Block, please refer to Figure 3 above. The process by which the self-attention module of the above-mentioned image encoder 14 processes at least one compressed image feature matrix is the same as the process by which the self-attention module of the above-mentioned first conditional encoder 111 processes the pixel rearrangement feature vector. The specific implementation can be found in the relevant description of the above-mentioned embodiment, and this application will not go into details here.

It should be noted that the above-mentioned image encoder 14 may include multiple convolution modules and multiple self-attention modules, and the number of convolution modules is one more than the number of self-attention modules. Different convolution modules perform different degrees of feature compression on the feature matrix, and different self-attention modules extract the correlation between pixel points in feature matrices of different dimensions.

Exemplarily, as shown in FIG9, it is a schematic diagram of the structure of an image encoder provided by some embodiments of the present application. In FIG9, the image encoder includes three convolution modules and two self-attention modules. The three convolution modules are convolution module 1 (ConvBlock1), convolution module 2 (ConvBlock2) and convolution module 3 (ConvBlock3). The two self-attention modules are self-attention module 1 including 4 layers of Transformer Block and self-attention module 2 including 6 layers of Transformer Block. In addition, the functions of convolution module 1, convolution module 2 and convolution module 3 are to reduce the dimension of the feature matrix. The number of channels of the convolution kernel is different. The number of channels of convolution module 1 is 4, the number of channels of convolution module 2 is 8, and the number of channels of convolution module 3 is 16.

Exemplarily, in Figure 9, the electronic device inputs the original image with a dimension of [3, 512, 512] into the convolution module 1 for feature compression to obtain a feature matrix with a dimension of [4, 256, 256], and then inputs the compressed image feature matrix with a dimension of [4, 256, 256] into the self-attention module 1 to obtain a feature matrix with a dimension of [4, 256, 256], and then inputs the feature matrix with a dimension of [4, 256, 256] into the convolution module 2 for further feature compression to obtain a compressed image feature matrix with a dimension of [8, 128, 128], and then inputs the feature matrix with a dimension of [8, 128, 128] into the self-attention module 12 to obtain a feature matrix with a dimension of [8, 128, 128], and then inputs the feature matrix with a dimension of [8, 128, 128] into the convolution module 3 for further feature compression to obtain an image feature matrix with a dimension of [16, 64, 64].

In this way, before the denoising module 12 performs denoising on the original image, the image encoder 14 performs feature compression on the original image, and then performs denoising on the image feature matrix obtained after feature compression, which can reduce the amount of data that the denoising module 12 performs denoising on and improve processing efficiency.

In some embodiments of the present application, the noise adding module 12 is specifically used to perform noise adding processing on at least one image feature matrix based on a Gaussian noise matrix and a mask image of at least one original image to obtain at least one noise image.

In some embodiments of the present application, a noise image corresponds to a mask image of an original image and an image feature matrix.

In some embodiments of the present application, the image encoder 14 inputs at least one image feature matrix into the noise adding module 12. The noise adding module 12 performs noise adding processing on the at least one image feature matrix based on the Gaussian noise matrix and the mask image of at least one original image through the above formula (1) and formula (2) to obtain at least one noise image. The specific implementation thereof can refer to the relevant description of the above embodiment, and the present application will not repeat it here.

In some embodiments of the present application, the image decoder 15 is used to decode at least one derivative image to obtain a decoded derivative image with a higher resolution.

Exemplarily, the above-mentioned image decoder 15 can be any decoder that can realize the decoding function, that is, map a low-dimensional latent vector to a high-dimensional output.

For example, the image decoder 15 may be a VAE decoder (VAE-Decoder).

Exemplarily, taking the dimension of the derived image as [16, 64, 64], after the derived image is decoded by the image decoder 15, a decoded derived image with a dimension of [3, 512, 512] can be obtained.

In some embodiments of the present application, the above-mentioned image decoder 15 may include a deconvolution module and a self-attention module. The derived image is input into the deconvolution module to increase the resolution of the features and obtain a derived image feature matrix. The derived image feature matrix is input into the self-attention module. Based on the correlation between each pixel point in the derived image feature matrix, the feature of the derived image feature matrix is extracted to obtain a decoded derivative image.

In some embodiments of the present application, the deconvolution module may include a deconvolution layer, a nonlinear function activation layer and a convolution layer.

It should be noted that the above deconvolution layer is used to increase the resolution of the input feature matrix. The above nonlinear function activation layer is used to perform nonlinear transformation on the input feature matrix. The above convolution layer is used to extract features from the input feature matrix.

In some embodiments of the present application, the size, step size and number of output channels of the convolution kernel in the deconvolution layer and the convolution layer are all defined in advance.

Exemplarily, as shown in Figure 10, it is a structural diagram of a deconvolution module provided by some embodiments of the present application. In Figure 10, the deconvolution module includes a deconvolution layer, two convolution layers and three nonlinear function activation layers. One deconvolution layer is a deconvolution layer 2_1 (Deconv2_1), the two convolution layers are convolution layer 2_2 (Conv2_2) and convolution layer 2_3 (Conv2_3), and the three nonlinear activation function layers are nonlinear function activation layer 2_1 (Relu2_1), nonlinear function activation layer 2_2 (Relu2_2) and nonlinear function activation layer 2_3 (Relu2_3). In addition, the convolution kernel size of the deconvolution layer 2_1 is 3x3, the stride is 2, and the number of output channels is 16. It is then connected to a nonlinear function activation layer 2_1. The nonlinear function activation layer 2_1 is then connected to a convolution layer 2_2. The convolution kernel size of the convolution layer 2_2 is 3x3, the stride is 1, and the number of output channels is 8. It is then connected to a nonlinear function activation layer 2_2. The nonlinear function activation layer 2_2 is then connected to a convolution layer 2_3. The convolution kernel size of the convolution layer 2_3 is 3x3, the stride is 1, and the number of output channels is 8. It is then connected to a nonlinear function activation layer 2_3.

Exemplarily, in FIG10 , taking the derivative image whose dimension is [16, 64, 64] as an example, the derivative image whose dimension is [16, 64, 64] is input into the deconvolution layer 2_1 for deconvolution operation, and a feature matrix whose dimension is [8, 128, 128] is obtained and output, and then the feature matrix is input into the nonlinear function activation layer 2_1 to perform nonlinear transformation on the value of each pixel point, and a feature matrix whose dimension is [8, 128, 128] is output, and then the feature matrix is input into the convolution layer 2_2 for feature extraction, and a feature matrix whose dimension is [8, 128, 128] and output it, then input the feature matrix into the nonlinear function activation layer 2_2 to perform nonlinear transformation on the value of each pixel point, and output a feature matrix with a dimension of [8, 128, 128], then input the feature matrix into the convolution layer 2_3 for further feature extraction, and obtain a feature matrix with a dimension of [8, 128, 128] and output it, then input the feature matrix into the nonlinear function activation layer 2_3 to perform nonlinear transformation on the value of each pixel point, and output a feature matrix with a dimension of [8, 128, 128].

In some embodiments of the present application, the above-mentioned image decoder may include multiple deconvolution modules and multiple self-attention modules, and the number of deconvolution modules is one more than the number of self-attention modules. In different deconvolution modules, the dimension of the feature matrix is increased to different degrees, and in different self-attention modules, the correlation relationship between pixel points in feature matrices of different dimensions is extracted.

Exemplarily, as shown in FIG. 11, it is a schematic diagram of the structure of an image decoder provided by some embodiments of the present application. In FIG. 11, the image decoder includes three deconvolution modules and two self-attention modules. The three deconvolution modules are respectively deconvolution module 1 (DeConvBlock1), deconvolution module 2 (DeConvBlock2) and deconvolution module 3 (DeConvBlock3). The two self-attention modules are respectively self-attention module 3 including 6 layers of Transformer Block and self-attention module 4 including 4 layers of Transformer Block. Moreover, the function of deconvolution module 1, deconvolution module 2 and deconvolution module 3 is to increase the feature resolution. There is only a difference in the number of channels of the convolution kernel. The number of channels of deconvolution module 1 is 16, the number of channels of deconvolution module 2 is 8, and the number of channels of deconvolution module 3 is 4.

Exemplarily, in Figure 11, the implementation process of the above-mentioned image decoder decoding the derivative image to obtain the decoded derivative image is as follows: the derivative image with a dimension of [16, 64, 64] is input into the deconvolution module 1 to increase the feature resolution, and a feature matrix with a dimension of [8, 128, 128] is obtained, and then the feature matrix with a dimension of [8, 128, 128] is input into the self-attention module 3 to obtain a feature matrix with a dimension of [8, 128, 128], and then the feature matrix with a dimension of [8, 128, 128] is input into the deconvolution module 2 to further increase the feature resolution, and a feature matrix with a dimension of [4, 256, 256] is obtained, and then the feature matrix with a dimension of [4, 256, 256] is input into the self-attention module 4 to obtain a feature matrix with a dimension of [4, 256, 256], and then the feature matrix with a dimension of [4, 256, 256] is input into the deconvolution module 3 to further increase the feature resolution, and a decoded derivative image with a dimension of [3, 512, 512] is obtained.

It should be noted that the self-attention module of the above-mentioned image decoder 15 includes multiple Tansformer Blocks, and the structural diagram of Tansformer Block is shown in the above-mentioned Figure 3. The process of processing the feature matrix of the self-attention module of the above-mentioned image decoder 15 is the same as the process of processing the pixel rearrangement feature vector by the self-attention module of the above-mentioned first conditional encoder 111. The specific implementation can refer to the relevant description of the above-mentioned embodiment, and this application will not repeat it here.

In this way, the image generation model includes an image encoder and an image decoder. The image encoder is used to compress the features of the original image before the denoising module performs denoising processing, thereby reducing the amount of calculation in the denoising process and the amount of data processing in the subsequent generation of derivative images, thereby improving processing efficiency. After the derivative image is obtained, it is decoded by the image decoder to obtain a decoded derivative image with higher resolution.

Exemplarily, as shown in FIG2 , the image generation model includes a conditional encoder 11, an image encoder 14, a noise adding module 12, a diffusion model 13 and an image decoder 15. The conditional encoder 11 includes a first conditional encoder 111 and a second conditional encoder 112. The second conditional encoder 112 includes a text encoder small and a text encoder big. Taking an original image, a description text of the original image and a mask image of the original image as an example, the data processing process of the image generation model is described: the electronic device inputs the original image into the first conditional encoder 111 for encoding to obtain an image feature information, inputs the description text into the text encoder small and the text encoder big for encoding respectively, and obtains two types of text feature information, collectively referred to as a text feature information, and then determines the conditional feature information according to the control condition, an image feature information and a text feature information, and the conditional encoder 11 inputs the conditional feature information into the diffusion model 13. The electronic device inputs the original image into the image encoder 14 for feature compression, obtains the image feature matrix and inputs it into the denoising module 12, the electronic device inputs the mask image into the denoising module 12, the denoising module 12 performs denoising processing on the image feature matrix based on the Gaussian noise matrix and the mask image to obtain a noise image, and the denoising module 12 inputs the noise image into the diffusion model 13. The diffusion model 13 generates a derivative image based on the conditional feature information obtained by the conditional encoder 11, the noise image obtained by the denoising module 12, and the noise vector, the diffusion model 13 inputs the derivative image into the image decoder 15, and the image decoder 15 decodes the derivative image to obtain a decoded derivative image.

The image generation model provided by some embodiments of the present application determines conditional feature information according to the type of image generation task to control the image generation model to perform different image generation tasks, and uses a Gaussian noise matrix and a mask image to perform noise processing on the original image, so that the image generation model can not only have the ability to generate images from noise, but also have the ability to predict, eliminate or modify the image content of the mask area corresponding to the mask image, so that the image generation model can adapt to different types of image generation tasks, such as text-generated images, image-generated images, text-plus-image-generated images, local image modification, image elimination, image expansion, etc. In this way, an image generation model has the ability to handle multiple types of image generation tasks, and there is no need to call multiple different models when handling different types of image generation tasks, thereby improving the efficiency of processing image generation tasks.

Some embodiments of the present application provide a training method for an image generation model, and the executing entity may be a training device for the image generation model. Exemplarily, the training device for the image generation model may be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The specific one may be determined based on actual usage requirements, and some embodiments of the present application do not limit this. The following takes the training device for the image generation model as an electronic device, and takes the electronic device executing the training method for the image generation model as an example, to exemplify the training method for the image generation model provided in some embodiments of the present application.

It should be noted that the image generation model trained by the image generation model training method provided in some embodiments of the present application is the image generation model described in the above embodiments.

Figure 12 is a flow chart of the training method of the image generation model provided by some embodiments of the present application. As shown in Figure 12, the training method of the image generation model provided by some embodiments of the present application may include the following steps 101 to 104.

Step 101: The electronic device obtains at least two training sample groups.

In some embodiments of the present application, each of the at least two training sample groups includes a sample image, description text of each sample image, and a mask image.

In some embodiments of the present application, the sample image may be an image including at least one entity, and the number of the sample images may be at least one.

Exemplarily, the sample image may include at least one of entities such as plants, animals, people, mountains, water, houses, computers, etc.

In some embodiments of the present application, the description text of the sample image may be used to describe the image theme of the sample image.

For example, the description text of the sample image may be “a man and a woman standing in front of a lawn, making a thumbs-up gesture”.

In some embodiments of the present application, the description text of the sample image may be used to describe the image theme of the sample image and the processing operation performed on the sample image.

For example, the description text of the sample image may be "A man and a woman stand in front of a lawn, making a thumbs-up gesture. Delete the man in the picture."

In some embodiments of the present application, the description text of the sample image may also be used to describe the image theme of the sample image and the image content of the mask area in the sample image.

For example, the description text of the sample image may be “a man and a woman standing in front of a lawn, making a thumbs-up gesture, and the picture contains a man, a woman, a balloon, a poster, and a hat.” It is understandable that the mask area of the sample image includes the image areas corresponding to the man, the woman, the balloon, the poster, and the hat in the sample image.

It should be noted that in some embodiments of the present application, the mask image of the sample image is used to represent the mask area in the sample image. It is understandable that after the mask image of the sample image is acquired, the mask area in the sample image can be determined.

It should be noted that the mask image of the sample image is in the same form as the mask image of the original image in the above embodiment. For the description of the mask image of the sample image, please refer to the relevant description of the above embodiment, and this application will not repeat it here.

In some embodiments of the present application, the electronic device can construct a training sample group using two methods: constructing by segmented entities and constructing by size information. The entity can be an object actually existing in the sample image, such as a person, an animal, a plant, a table, a computer, etc., or an object of interest actually existing in the sample image, such as the subject of the sample image.

It should be noted that the electronic device constructed by segmenting entities can be understood as entity recognition and segmentation of the sample image, using the image area of each entity obtained by segmenting the sample image as a mask area, and generating a mask image of the sample image based on the mask area, thereby constructing a training sample group.

It should be noted that the electronic device constructed according to the size information can be understood as not considering which entities are in the sample image, but randomly determining the mask area in the sample image according to the size information, and generating a mask image of the sample image based on the determined mask area, thereby constructing a training sample group.

In some embodiments of the present application, the electronic device may construct a training sample group in a manner of constructing by segmenting entities.

Exemplarily, in combination with FIG. 12 , as shown in FIG. 13 , before the above-mentioned step 101 , the training method of the image generation model provided by some embodiments of the present application may further include the following steps 201 to 204 .

Step 201: The electronic device obtains at least two sample images, image description information of each sample image, and annotation information of at least two mask regions of each sample image.

In some embodiments of the present application, the image description information of the sample image is used to describe the image theme of the sample image.

In some embodiments of the present application, at least two mask regions of each sample image are obtained by predicting the mask regions of each sample image using a segmentation model and segmenting each sample image based on the prediction results.

In some embodiments of the present application, the open source segmentation dataset includes tens of millions of images and billions of segmented mask areas, which are rich in entities and have high information density. Therefore, the electronic device can obtain sample images, image description information of sample images, and annotation information of multiple mask areas from the open source segmentation dataset. In this case, the multiple mask areas corresponding to the sample images in the segmentation dataset are obtained by inputting the sample images into the Segment Anything Model (SAM) and then performing entity segmentation by SAM, and the image description information of the sample images is carried by the sample images.

In some embodiments of the present application, the annotation information of each mask region includes at least one of the following: the area of each mask region, the prediction accuracy of each mask region, and the prediction stability of each mask region.

In some embodiments of the present application, the prediction accuracy represents the prediction accuracy of the mask region by the segmentation model, and the prediction stability represents the prediction stability of the mask region by the segmentation model.

Exemplarily, the prediction accuracy of the mask area is the degree of overlap between the predicted box of the mask area and the true box, which indicates the accuracy of SAM in predicting the mask area; the prediction stability of the mask area is the confidence of SAM in predicting the mask area, which indicates the stability of SAM in predicting the mask area.

Step 202: The electronic device determines, based on the annotation information of at least two mask areas of each sample image, a first mask area whose annotation information satisfies a filtering condition from among the at least two mask areas of each sample image.

In some embodiments of the present application, the above-mentioned filtering conditions include at least one of the following: the area of the mask area is within a preset range, the prediction accuracy of the mask area is greater than the accuracy threshold, and the prediction stability of the mask area is greater than the stability threshold.

It should be noted that the above-mentioned preset range, accuracy threshold and stability threshold can be set or adjusted according to actual needs. For example, the above-mentioned preset range can be 5%-45%, the above-mentioned accuracy threshold can be 0.8, and the above-mentioned stability threshold can be 0.9. Some embodiments of the present application do not limit this.

In some embodiments of the present application, the area of the mask region being within a preset range means that the ratio of the area of the mask region to the area of the sample image is within a preset range.

Exemplarily, taking the case where the annotation information of at least two mask regions includes the area of each mask region, the prediction accuracy of each mask region, and the prediction stability of each mask region as an example, assuming that the filtering conditions include that the area of the mask region is within the range of 5%-45%, the prediction accuracy of the mask region is greater than 0.8, and the prediction stability of the mask region is greater than 0.9, the electronic device can select a mask region whose area accounts for 5%-45% of the area of the sample image, and whose prediction accuracy is greater than 0.8, and whose prediction stability is greater than 0.9 from the at least two mask regions as the first mask region. The number of the first mask region can be at least one.

In this way, the electronic device filters out the smaller mask area, reducing the training data that is not helpful for model training. Filtering out the mask area with too low prediction stability and too low prediction accuracy can reduce low-quality training samples, so that the prediction results in the model training process are more stable, reducing the difficulty of model training, accelerating model convergence, and improving model training efficiency.

Step 203: The electronic device adds the description information of the first mask area of each sample image to the image description information of each sample image to obtain a description text of each sample image.

In some embodiments of the present application, the description information of the first mask area is used to describe the image content corresponding to the first mask area in the sample image.

Exemplarily, the description information of the first mask area may include the type of entity, such as man, woman, table, computer, etc., and may also include the appearance of the entity, such as white cat, black dog, square table, round table, etc.

In some embodiments of the present application, the annotation information of the above-mentioned at least two mask areas may also include description information of each mask area. The electronic device can obtain the description information from the annotation information of the first mask area, add the description information to the image description information, and obtain the description text of the sample image. The description text can then be used to describe the image theme of the sample image and the image content of the first mask area corresponding to the sample image.

Step 204: The electronic device constructs each sample image, the description text of each sample image, and the mask image of each sample image into a training sample group, thereby obtaining at least two training sample groups.

In some embodiments of the present application, the mask image of each sample image is a mask image corresponding to the first mask area in each sample image.

In some embodiments of the present application, after determining the first mask area, the electronic device can generate a first matrix, the number of element points in the first matrix is the same as the number of pixel points in the sample image, and the arrangement of the element points in the first matrix is the same as the arrangement of the pixel points in the sample image, determine the element points corresponding to the first mask area in the first matrix, mark the values of these element points as 1, and mark the values of other element points in the first matrix as 0, and the final first matrix is the mask image of the sample image.

In this way, the electronic device determines at least two mask areas in the sample image by performing entity segmentation on the sample image, and filters out the mask areas of the at least two mask areas that are not very meaningful for model training, such as mask areas with small areas, inaccurate predictions, or unstable predictions, and retains the first mask area, thereby reducing the noise of the training data, reducing the difficulty of model training, and thereby improving the training efficiency of the model, and making the prediction results in the model training process more stable, making the model convergence more stable, and adding the description information of the first mask area to the image description information to obtain the description text of the sample image, so that the model can learn to predict the image content of the first mask area according to the description text, thereby improving the model performance.

Furthermore, for a sample image, when the number of first mask areas is greater than a quantity threshold, X third mask areas can be randomly selected from the first mask areas, and then the description information of the third mask areas of the sample image is added to the image description information of the sample image to obtain a description text of the sample image, and generate a mask image of the sample image, which corresponds to the X third mask areas of the sample image, and the sample image, the description text of the sample image and the mask image of the sample image are constructed as a training sample group.

It should be noted that X is an integer greater than or equal to 1, for example, X is 5.

It should be noted that the above quantity threshold may be set by the electronic device by default, or may be set by the user according to actual needs, and this application does not limit this. For example, the quantity threshold may be 20.

In this way, after the electronic device uses the mask image to perform noise processing on the sample image, the obtained noised image includes more original image information, so that the model can learn more image information of the sample image and improve the learning effect of the model.

It is worth noting that when the number of the first mask regions is greater than the number threshold, the electronic device can randomly select X third mask regions from the first mask regions before each iterative training, and generate a training sample group through the above steps. In addition, the X third mask regions selected in different iterative trainings can be different, so that the model can learn the ability to predict the image content of the mask region under multiple mask combinations.

In some embodiments of the present application, the electronic device may construct a training sample group in a manner constructed according to size information.

Exemplarily, in combination with FIG. 12 , as shown in FIG. 14 , before the above-mentioned step 101 , the training method of the image generation model provided by some embodiments of the present application also includes the following steps 301 to 303 .

Step 301: The electronic device obtains at least two sample images and a description text of each sample image.

In some embodiments of the present application, the electronic device may obtain sample images and description texts of the sample images from an open source image collection.

Step 302: The electronic device generates a second mask area in each sample image according to the size range and the size information of each sample image.

In some embodiments of the present application, the size range may include a height range and a width range of the second mask area. The height range is the range of the area height of the second mask area to the height of the sample image, and the width range is the range of the area width of the second mask area to the width of the sample image.

Exemplarily, the height may range from 5% to 15%, and the width may range from 10% to 20%.

In some embodiments of the present application, the size information of the sample image may include the height and width of the sample image.

In some embodiments of the present application, after acquiring the sample image, the electronic device can select any percentage in the height range, multiply the percentage by the height of the sample image, and obtain the region height of the second mask region, and select any percentage in the width range, multiply the percentage by the width of the sample image, and obtain the region width of the second mask region. Then, the electronic device can determine a region with a height equal to the region height and a width equal to the region width at any position of the sample image as the second mask region.

Exemplarily, the electronic device may determine, in the sample image, a first region whose width is the width of the sample image and whose height is the region height starting from the upper edge of the sample image, and determine, in the sample image, a second region whose width is the width of the sample image and whose height is the region height starting from the lower edge of the sample image, and determine, in the sample image, a third region whose height is the height of the sample image and whose width is the region width starting from the left edge of the sample image, and determine, in the sample image, a fourth region whose height is the height of the sample image and whose width is the region width starting from the right edge of the sample image, and combine the first region, the second region, the third region and the fourth region to obtain a second mask region.

For example, taking the case where both the height and width of the sample image are 512, the width range is 10% to 20%, and the height range is 5% to 15%, assuming that the height range selected by the electronic device is 10% and the width range is 15%, it can be determined that the region height is 51 and the region width is 76. The electronic device can determine a first region with a width of 512 and a height of 51 in the sample image starting from the upper edge of the sample image, and a second region with a width of 512 and a height of 51 in the sample image starting from the lower edge of the sample image, and a third region with a height of 512 and a width of 76 in the sample image starting from the left edge of the sample image, and a fourth region with a height of 512 and a width of 76 in the sample image starting from the right edge of the sample image, and the first region, the second region, the third region and the fourth region are combined to obtain a second mask region.

Step 303: The electronic device constructs each sample image, the description text of each sample image, and the mask image of each sample image into a training sample group, thereby obtaining at least two training sample groups.

In some embodiments of the present application, the mask image of each sample image is a mask image of the second mask area in each sample image.

In some embodiments of the present application, after determining the second mask area, the electronic device can generate a second matrix, the number of element points in the second matrix is the same as the number of pixel points in the sample image, and the arrangement of the element points in the second matrix is the same as the arrangement of the pixel points in the sample image, determine the element points corresponding to the second mask area in the second matrix, mark the values of these element points as 1, and mark the values of other element points in the second matrix as 0, and the final second matrix is the mask image of the sample image.

Exemplarily, as shown in FIG15 , it is a schematic diagram of a mask image provided by some embodiments of the present application. In FIG15 , the mask image is represented in the form of an image, wherein the black portion represents the second mask area, the white portion represents the unmasked area, M_h represents the height of the second mask area, and M_w represents the width of the second mask area.

In this way, the electronic device generates a second mask area in the sample image according to the size information without considering the entity, and generates a mask image of the sample image. The form of the mask image of the determined sample image is richer, that is, richer training samples can be obtained, so that the model can learn the ability to predict images under various mask situations, thereby improving the performance of the model.

In some embodiments of the present application, the two methods of constructing training sample groups mentioned above can be used separately or in combination. For example, during the model training process, 50% of the training sample groups are constructed by segmenting entities, and 50% of the training sample groups are constructed by size information. In this way, richer training data can be obtained, so that the model can learn the ability to generate or process images in various situations, thereby improving model performance.

Step 102: The electronic device encodes at least one sample image in at least two training sample groups to obtain at least one item of image feature information, and encodes the description text of each sample image to obtain at least one item of text feature information.

In some embodiments of the present application, one item of image feature information in the at least one item of image feature information is used to characterize a sample image. One item of text feature information in the at least one item of text feature information is used to characterize a description text of a sample image.

In some embodiments of the present application, the above step 102 can be specifically implemented by following the steps 401 to 403.

Step 401: The electronic device divides each sample image into N sample image blocks through the image segmentation module in the first conditional encoder, and rearranges the pixels of each sample image block into a one-dimensional vector to obtain N one-dimensional vectors of each sample image, and concatenates the N one-dimensional vectors of each sample image to obtain a pixel rearranged image of each sample image.

Step 402: The electronic device extracts feature information of each pixel in each pixel rearrangement image through a mapping layer in the first conditional encoder to obtain at least one pixel rearrangement feature vector.

Step 403: The electronic device uses the self-attention module in the first conditional encoder to extract features from each pixel rearrangement feature vector based on the correlation between pixels in each pixel rearrangement image, so as to obtain image feature information of each sample image.

It should be noted that the first conditional encoder in steps 401 to 403 has the same structure as the first conditional encoder 111 in the above embodiment (as shown in FIG. 4 ), and the process of encoding the image is also the same. Therefore, the specific implementation process of steps 401 to 403 can refer to the relevant description of the above embodiment, and this application will not repeat it here.

Exemplarily, as shown in FIG4 , the first conditional encoder includes an image segmentation module, a mapping layer, and a self-attention module. The specific implementation of the first conditional encoder encoding a sample image to obtain an image feature information includes: the electronic device inputs a sample image with a resolution of 512×512×3 into the image segmentation module of the first conditional encoder, divides the sample image into N sample image blocks through the image segmentation module, and rearranges the pixels of each sample image block into a one-dimensional vector to obtain N one-dimensional vectors of the sample image, and splices the N one-dimensional vectors of the sample image to obtain a pixel rearranged image with a resolution of 256×768 for the sample image, inputs the mapping layer, performs linear transformation processing on each pixel in the pixel rearranged image through the mapping layer to obtain feature information of each pixel, splices and combines feature information of multiple pixels in the pixel rearranged image to obtain a pixel rearranged feature vector with a dimension of 256×768, inputs the pixel rearranged feature vector into the self-attention module, and processes the pixel rearranged feature vector in the self-attention module in sequence through the self-attention operator, the addition & normalization operator, the forward feedback operator, and the addition & normalization operator to obtain image feature information of the sample image with a dimension of 256×768.

In this way, the electronic device extracts features of the sample image through the first conditional encoder to obtain image feature information of the sample image, and uses it in subsequent model training, so that the model can learn the image feature information of the image and have the ability to process images.

In some embodiments of the present application, the above step 102 can be specifically implemented by following the steps 501 to 502.

Step 501: The electronic device divides each description text into at least two words through a word segmenter in a second conditional encoder, and encodes each word segment to obtain a text feature vector of each description text.

Step 502: The electronic device uses the self-attention module in the second conditional encoder to extract features from the text feature vector of each description text based on the association between the word segments in each description text, so as to obtain text feature information of each description text.

It should be noted that the second conditional encoder in step 501 to step 502 has the same structure as the second conditional encoder 112 in the above embodiment (as shown in FIG. 5 ), and the process of encoding the text is also the same. Therefore, the specific implementation process of step 501 to step 502 can refer to the relevant description of the above embodiment, and this application will not repeat it here.

Exemplarily, as shown in FIG5 , the second conditional encoder includes a word segmenter and a self-attention module. The specific implementation of the second conditional encoder encoding the description text of a sample image to obtain a text feature information includes: the electronic device inputs the description text into the word segmenter of the second conditional encoder, divides the description text into at least two word segments through the word segmenter, and encodes each word segment to obtain a text feature vector of the description text with a dimension of 128×768, which is input into the self-attention module, and the text feature vector is processed by the self-attention operator, the addition & normalization operator, the forward feedback operator, and the addition & normalization operator in the self-attention module in turn to obtain text feature information of the description text with a dimension of 128×768.

In this way, the description text is feature extracted through the second conditional encoder to obtain text feature information of the description text, and used in subsequent model training, so that the model can learn the text feature information, which facilitates the model to learn to more accurately process the image generation task based on the description text.

Step 103: The electronic device determines at least one item of conditional feature information based on the control condition, at least one item of image feature information and at least one item of text feature information.

In some embodiments of the present application, the above control conditions are used to indicate the type of image generation task, and the above conditional feature information is determined according to the type of image generation task.

In some embodiments of the present application, the above step 103 can be specifically implemented through the following step 601.

Step 601: When the type of image generation task indicated by the control condition is a text-image task, the electronic device determines at least one item of text feature information as at least one item of conditional feature information; when the type of image generation task indicated by the control condition is an image processing task, the electronic device determines at least one item of image feature information and at least one item of text feature information as at least one item of conditional feature information.

In some embodiments of the present application, the image processing task includes at least one of the following: an image-to-image task, a text-to-image task, an image elimination task, an image local modification task, and an image expansion task.

It should be noted that the specific implementation process of determining at least one conditional feature information in step 601 is the same as the implementation process of the above embodiment. The specific implementation can refer to the relevant description of the above embodiment, and this application will not repeat it here.

In this way, by using control conditions, the image generation model can determine the conditional feature information and perform subsequent processing according to the type of image generation task, so that a trained image generation model can have the ability to handle multiple types of image generation tasks, thereby improving the training efficiency of the image generation model and the processing efficiency of the image generation task.

Step 104: The electronic device trains the initial model based on at least one sample image, at least one mask image of the sample image, and at least one conditional feature information to obtain an image generation model.

In some embodiments of the present application, the initial model is an untrained image generation model whose structure is the same as that of the above-mentioned image generation model, but the model parameters are initial parameters and need to be trained before it can have the ability to handle various types of image generation tasks.

In some embodiments of the present application, the above step 104 can be specifically implemented by following the steps 701 to 702.

Step 701: The electronic device performs noise addition processing on at least one sample image based on a Gaussian noise matrix and a mask image of at least one sample image to obtain at least one noise image.

In some embodiments of the present application, the Gaussian noise matrix is a noise matrix that obeys a normal distribution and can be obtained by random sampling of an electronic device.

In some embodiments of the present application, a noise image corresponds to a mask image of a sample image and a sample image.

In some embodiments of the present application, the above step 701 can be specifically implemented by following the steps 701a to 701b.

Step 701a: The electronic device performs weighted summation on the Gaussian noise matrix and the mask image of at least one sample image to obtain at least one noise matrix.

Exemplarily, the electronic device can calculate the noise matrix using the above formula (1).

Exemplarily, the mask image used when determining the noise matrix can be randomly determined in the above two different ways, and the generated noise matrix can also include two different forms. As shown in Figure 16, it is a schematic diagram of two methods for determining noise matrices provided by some embodiments of the present application. In Figure 16, the electronic device multiplies the Gaussian noise matrix (GNM) with a dimension of [16, 64, 64] by Weight, and multiplies the mask image (Mask) with a dimension of [16, 64, 64] determined by the segmentation entity construction method by (1-Weight), and then sums the results of the two multiplications to obtain a noise matrix z' with a dimension of [16, 64, 64]; the electronic device multiplies the Gaussian noise matrix (GNM) with a dimension of [16, 64, 64] by Weight, and multiplies the mask image (Mask) with a dimension of [16, 64, 64] determined by the size information construction method by (1-Weight), and then sums the results of the two multiplications to obtain a noise matrix z' with a dimension of [16, 64, 64].

In this way, multiple noise adding methods can be combined to obtain multiple noise adding matrices. In the process of model training, the model can learn to reconstruct images from a variety of different noises, thereby improving the performance of the model.

Step 701b: The electronic device performs weighted summation on at least one noise matrix and at least one sample image to obtain at least one noise image.

In some embodiments of the present application, each noisy image corresponds to a noise matrix and a sample image.

Exemplarily, the electronic device can calculate the noise image using the above formula (2).

It should be noted that the specific implementation of the above steps 701a to 701b is the same as the implementation process of adding noise to the original image to obtain a noisy image in the above embodiment. The specific implementation can be found in the relevant description of the above embodiment, and this application will not repeat it here.

In this way, the electronic device performs noise processing on the sample image by using only a Gaussian noise matrix or only a mask image or a combination of a Gaussian noise matrix and a mask image, and the mask image is obtained by the above-mentioned two methods of constructing according to the segmentation entity and constructing according to the size information. There are many forms of the noise matrix, and there are also many forms of the noise image obtained by noisy the sample image. Therefore, the model can not only learn the ability to reconstruct images from various noises, but also learn the ability to predict the image content of the mask area under various masking methods, thereby improving the performance of the model.

Step 702: The electronic device trains an initial model based on at least one conditional feature information and at least one noise image to obtain an image generation model.

In some embodiments of the present application, each noise image corresponds to one item of conditional feature information.

In some embodiments of the present application, the above-mentioned image generation model is used to process image generation tasks.

In some embodiments of the present application, the image generation task includes a text image task and an image processing task. The image processing task includes at least one of the following: local image modification, local image elimination, and image expansion.

In some embodiments of the present application, the above step 702 can be specifically implemented by following the steps 801 to 804.

Step 801: The electronic device obtains feature information of at least one noise image through a diffusion model, and fuses at least one conditional feature information and feature information of at least one noise image to obtain at least one fused feature information.

In some embodiments of the present application, each fusion feature information corresponds to one conditional feature information and one feature information of a noise image.

It should be noted that the diffusion model in the above step 801 has the same structure as the diffusion model 103 in the above embodiment. For the specific structure, reference may be made to the relevant description of the above embodiment, which will not be described in detail in this application.

In some embodiments of the present application, before the above step 801, the training method of the image generation model provided by some embodiments of the present application also includes the following steps 901 to 902.

Step 901: The electronic device divides each noise image into M noise image blocks through the image segmentation module in the diffusion model, where M is an integer greater than 1.

Step 902: The electronic device extracts feature information of each noise image block through a linear transformation operator in a diffusion model, and adds a position coding vector to obtain feature information of each noise image.

Exemplarily, taking a noise image as an example, the electronic device inputs the noise image into the image segmentation module in the diffusion model, divides the noise image into M noise image blocks through the image segmentation module, and then inputs the linear transformation operator. The electronic device extracts the feature information of each of the M noise image blocks through the linear transformation operator, and generates a position coding vector for each noise image block. According to the position of each noise image block in the noise image, the feature information of the M noise image blocks is spliced, and the position coding vector is added to obtain the feature information of the noise image.

It should be noted that the specific implementation of the above steps 901 to 902 is the same as the implementation process of extracting the feature information of the noise image in the above embodiment. The specific implementation can be found in the relevant description of the above embodiment, and this application will not repeat it here.

In this way, the electronic device extracts the characteristic information of the noise image through the image segmentation module and the linear transformation operator, and the diffusion model combines the characteristic information of the noise image and the conditional characteristic information to predict the noise matrix, thereby improving the prediction accuracy.

In some embodiments of the present application, the above step 801 can be specifically implemented by following the steps 801a to 801b.

Step 801a: The electronic device performs linear transformation processing on at least one item of conditional feature information by using a linear transformation operator in a diffusion model to obtain at least one item of conditional feature information after linear transformation.

Step 801b: The electronic device splices at least one item of conditional feature information after linear transformation and feature information of at least one noise image through a splicing operator in the diffusion model to obtain at least one item of fused feature information.

In some embodiments of the present application, one fused feature information in the at least one fused feature information corresponds to a conditional feature information after linear transformation and feature information of a noise image.

Exemplarily, the electronic device performs linear transformation processing on each conditional feature information through the above-mentioned linear transformation operator, obtains the conditional feature information after linear transformation and inputs it into the above-mentioned splicing operator, and splices the conditional feature information after linear transformation and the feature information of the noise image in dimension through the above-mentioned splicing operator to obtain fused feature information.

It should be noted that the specific implementation of the above steps 801a to 801b is the same as the implementation process of obtaining the fusion feature information in the above embodiment. The specific implementation can be found in the relevant description of the above embodiment, and this application will not repeat it here.

In this way, the electronic device maps the conditional feature information to the same feature space as the feature information of the noise image through a linear transformation operator, and then splices the conditional feature information after the linear transformation with the feature information of the noise image through a splicing operator to obtain fused feature information, providing a data basis for subsequent processing of the diffusion model.

Step 802: The electronic device determines a noise matrix of at least one noise image by using a diffusion model based on at least one fusion feature information and at least one noise vector, where each noise vector represents a noise level of a noise image.

In some embodiments of the present application, the above step 802 can be specifically implemented by following the steps 802a to 802c.

Step 802a, the electronic device performs feature extraction on at least one noisy vector through the diffusion self-attention module in the diffusion model to obtain at least one set of scaling factors, and based on the at least one set of scaling factors, performs feature scaling on at least one item of fused feature information to obtain at least one item of scaled feature information, each item of scaled feature information corresponds to a set of scaling factors and one item of fused feature information.

It should be noted that the diffuse self-attention module in the above step 802a has the same structure as the diffuse self-attention module in the above embodiment (as shown in Figure 6), and the data processing process is also the same. The structure of the diffuse self-attention module and the specific implementation process of step 802a can be found in the relevant description of the above embodiment, and this application will not repeat them here.

Step 802b: The electronic device normalizes at least one item of scaling feature information using a layer normalization operator in a diffusion model to obtain at least one item of noise feature information.

Step 802c: The electronic device performs channel reduction processing on at least one item of noise feature information by using a convolution operator in a diffusion model to obtain a noise matrix of at least one noise image.

It should be noted that the specific implementation of the above steps 802a to 802c is the same as the implementation process of determining the noise matrix of the noise image in the above embodiment. The specific implementation can refer to the relevant description of the above embodiment, and this application will not repeat it here.

Exemplarily, as shown in FIG7, the implementation process of the electronic device determining the noise matrix through the diffusion model includes: the electronic device performs linear transformation processing on the conditional feature information through the linear transformation operator to obtain the conditional feature information after the linear transformation. The electronic device divides the noise image into M image blocks through the image segmentation module and inputs the linear transformation operator, performs feature stealing on the M image blocks through the linear transformation operator to obtain the feature information of each image block, and then adds the position encoding vector to obtain the feature information of the noise image. The electronic device inputs the conditional feature information after the linear transformation and the feature information of the noise image into the splicing operator for splicing, obtains the fused feature information, inputs the diffusion self-attention module, extracts the features of the noise vector through the diffusion self-attention module, obtains a set of scaling factors, and performs scaling processing on the fused feature information according to the set of scaling factors to obtain the scaled feature information input to the layer normalization operator, performs reshape operation and Layer Norm operation on the scaled feature information through the layer normalization operator, obtains the noise feature information input to the convolution operator, reduces the number of channels of the noise feature information through the convolution operator, and obtains the noise matrix of the noise image.

In this way, the electronic device fuses the conditional feature information and the feature information of the noise image through a diffusion model, and then adds noise to the fused feature information with a noise vector until a noise matrix that is completely noise is obtained. Moreover, due to the addition of the conditional feature information, the prediction of the noise matrix combines the text feature information or the text feature information and the image feature information, so that the model can learn the ability to predict the noise matrix when there is only text or text and image.

Step 803: The electronic device determines a noise loss value of at least one sample image based on a noise matrix of at least one noise image, a Gaussian noise matrix, and a mask image of at least one sample image.

In some embodiments of the present application, the above step 803 can be specifically implemented by following the steps 803a to 803b.

Step 803a: The electronic device determines at least one first loss value based on the Gaussian noise matrix and the noise matrix of at least one noise image, and determines at least one second loss value based on the mask image of at least one sample image and the noise matrix of at least one noise image.

Step 803b: The electronic device performs weighted summation of at least one first loss value and at least one second loss value to obtain a noise loss value of at least one sample image.

In some embodiments of the present application, the electronic device may determine the noise loss value by the following formula (3):

Loss＝α*MSE(GNM, PN ₁ [16,:,:])+(1-α)*MSE(Mask, PN ₂ [16,:,:]) (3)

In formula (3), Loss represents the noise loss value, GNM represents the Gaussian noise matrix, Mask represents the mask image, PN ₁ [16,:,:] represents the matrix composed of the first 16 channels of the noise matrix, PN ₂ [16,:,:] represents the matrix composed of the last 16 channels of the noise matrix, α represents the weight, MSE(GNM, PN ₁ [16,:,:]) represents the first loss value, and MSE(Mask, PN ₂ [16,:,:]) represents the second loss value.

It should be noted that the above-mentioned first loss value enables the model to reconstruct the image from the Gaussian noise matrix, and the above-mentioned second loss value increases the supervision signal, allowing the model to predict, eliminate or modify the image content of the mask area, so that the model has the ability to locally modify, locally eliminate and expand the image.

Exemplarily, as shown in Figure 17, it is a schematic diagram of loss value calculation provided by some embodiments of the present application. In Figure 17, the dimension of the noise matrix is [32, 64, 64], which is divided into two matrices with a dimension of [16, 64, 64]. The two matrices are respectively a matrix composed of the first 16 channels of the noise matrix and a matrix composed of the last 16 channels. The first loss value L1 is determined based on the matrix with a dimension of [16, 64, 64] composed of the first 16 channels and the Gaussian noise matrix with a dimension of [16, 64, 64]. The second loss value L2 is determined based on the matrix with a dimension of [16, 64, 64] composed of the last 16 channels and the mask image with a dimension of [16, 64, 64].

In this way, the electronic device reconstructs the loss function and adjusts the model parameters based on the loss function, so that the model can not only learn the ability to reconstruct images from noise, but also learn the ability to locally modify, locally eliminate or expand the image.

Step 804: The electronic device adjusts the model parameters of the initial model based on the noise loss value of at least one sample image to obtain an image generation model.

In some embodiments of the present application, the model parameters of the initial model may include parameters of each module in the initial model and each layer in each module.

In some embodiments of the present application, after the electronic device adjusts the model parameters of the initial model based on the noise loss value, it continues to use the training sample group to iteratively train the initial model until the number of iterations is greater than the number threshold, or the loss value calculated after a certain iteration is less than the loss value threshold. The training can be stopped to obtain the image generation model.

It should be noted that the above number threshold and loss value threshold can be preset or set according to actual needs, and this application does not limit this. For example, the number threshold can be 100, and the loss value threshold can be 0.5.

In this way, the electronic device trains the initial model based on the conditional feature information and the noise image. Since the Gaussian noise matrix and the mask image are added to the noise image, during the model training process, the model can not only learn to generate images from noise, but also learn to predict, eliminate or modify the image content of the image area corresponding to the mask image. Moreover, since the conditional feature information is generated according to the control conditions, image feature information and text feature information, the conditional feature information is combined with the noise image, so that the model can also learn to generate images according to text, generate images according to images or process images, generate images according to text and images or process images, and predict, eliminate or modify the image content of the mask area according to text, thereby enabling the trained image generation model to have the ability to handle various types of image generation tasks.

In this way, the electronic device performs noise processing on at least one sample image by using a Gaussian noise matrix and a mask image of at least one sample image to obtain at least one noise image. Each noise image includes not only the image information of the sample image, but also the noise information and the image information of the mask image. Therefore, the initial model is trained using at least one noise image and at least one conditional feature information, so that the final initial model can not only learn to generate images from noise, but also learn the ability to handle various image generation tasks. There is no need to train different models for different types of image generation tasks, which reduces the processing burden of the electronic device and improves the efficiency of model training.

In some embodiments of the present application, the larger the dimension, the greater the amount of calculation. Since the dimension of the sample image is usually relatively large, while the dimension of the Gaussian noise matrix may be relatively small, the sample image can be feature compressed first, and then the feature compressed image can be noised according to the Gaussian noise matrix and the mask image. This can reduce the amount of calculation required for the electronic device to perform noise addition processing, reduce resource usage, and improve processing efficiency.

In some embodiments of the present application, the above step 104 can be specifically implemented by following steps 901 to 903.

Step 901: The electronic device performs feature compression on at least one sample image through an image encoder to obtain at least one image feature matrix.

In some embodiments of the present application, one image feature matrix corresponds to one sample image.

In some embodiments of the present application, the number of pixels in the above-mentioned image feature matrix is less than the number of pixels in a sample image.

For example, taking a sample image whose dimension is [3, 512, 512] as an example, the electronic device may perform feature compression on the sample image to obtain an image feature matrix whose dimension is [16, 64, 64].

In some embodiments of the present application, the above step 901 can be specifically implemented by following the steps 901a to 901b.

Step 901a: The electronic device performs feature compression on at least one sample image through a convolution module in an image encoder to obtain at least one compressed image feature matrix.

In some embodiments of the present application, the number of pixels in each compressed image feature matrix is greater than the number of pixels in each image feature matrix.

Step 901b: The electronic device uses a self-attention module in an image encoder to extract features from each compressed image feature matrix based on the correlation between pixels in each compressed image feature matrix to obtain at least one image feature matrix.

Exemplarily, as shown in Figure 9, the above-mentioned image encoder includes three convolution modules and two self-attention modules, the three convolution modules are convolution module 1, convolution module 2 and convolution module 3, and the two self-attention modules are self-attention module 1 including 4 layers of Transformer Block and self-attention module 2 including 6 layers of Transformer Block. The electronic device performs feature compression on a sample image through an image encoder to obtain an image feature matrix. The specific implementation includes: the electronic device inputs the sample image with a dimension of [3, 512, 512] into the convolution module 1 for feature compression to obtain a feature matrix with a dimension of [4, 256, 256], then inputs the compressed image feature matrix with a dimension of [4, 256, 256] into the self-attention module 1 to obtain a feature matrix with a dimension of [4, 256, 256], then inputs the feature matrix with a dimension of [4, 256, 256] into the convolution module 2 for further feature compression to obtain a compressed image feature matrix with a dimension of [8, 128, 128], then inputs the feature matrix with a dimension of [8, 128, 128] into the self-attention module 12 to obtain a feature matrix with a dimension of [8, 128, 128], then inputs the feature matrix with a dimension of [8, 128, 128] into the convolution module 3 for further feature compression to obtain an image feature matrix with a dimension of [16, 64, 64] of the sample image.

It should be noted that the specific implementation of the above steps 901a to 901b is the same as the implementation process of determining the image feature matrix of the original image in the above embodiment. The specific implementation can be found in the relevant description of the above embodiment, and this application will not repeat it here.

In this way, before the electronic device performs noise processing on the sample image, it compresses the features of the sample image through the convolution module in the image encoder, and further extracts features according to the correlation between the pixels through the self-attention module in the image encoder, so that the final image feature matrix not only has a small amount of data, but also can characterize the correlation between different pixels in the sample image, and can better characterize the sample image, that is, the amount of data for noise processing is reduced without destroying the image information of the sample image, thereby improving the processing efficiency of the electronic device.

Step 902: The electronic device performs noise addition processing on the at least one image feature matrix based on the Gaussian noise matrix and the mask image of the at least one sample image to obtain at least one noise image.

In some embodiments of the present application, a noise image corresponds to a mask image of an original image and an image feature matrix.

In some embodiments of the present application, the electronic device performs noise processing on at least one image feature matrix based on the Gaussian noise matrix and the mask image of at least one sample image through the above formula (1) and formula (2) to obtain at least one noise image. The specific implementation thereof can be referred to the relevant description of the above embodiment, and the present application will not repeat it here.

Step 903: The electronic device trains the initial model based on the at least one conditional feature information and the at least one noise image to obtain an image generation model.

It should be noted that the specific implementation of the above step 903 is the same as the implementation process of the above step 701. The specific implementation of the above step 903 can refer to the relevant description of the above step 701, and this application will not repeat it here.

In this way, before the electronic device performs noise processing on the sample image, it first performs feature compression on the sample image to reduce the number of pixels in the sample image, reduce the amount of calculation when adding noise to the sample image, and obtain a noise image with a smaller data volume, thereby reducing the amount of data in the subsequent model training process, reducing the resource usage of the electronic device, and improving the processing efficiency of the electronic device.

In this way, the electronic device uses the above-mentioned training method of the image generation model to determine the conditional feature information according to the type of image generation task, so as to train the image generation model using different types of image generation tasks, and use the Gaussian noise matrix and the mask image to perform noise processing on the sample image, so that the image generation model can not only have the ability to generate images from noise, but also have the ability to predict, eliminate or modify the image content of the mask area corresponding to the mask image, so that the image generation model can learn the ability to handle different types of image generation tasks. In this way, there is no need to train a corresponding model for each image generation task. An image generation model trained by the above method has the ability to handle multiple types of image generation tasks, which not only improves the efficiency of model training, but also improves the efficiency of processing image generation tasks.

It is worth noting that the above descriptions are all about the training process of the image generation model. The following describes the specific implementation process of using the image generation model to handle different image generation tasks after the image generation model is trained.

In some embodiments of the present application, after the above step 104, the training method of the image generation model provided by some embodiments of the present application may further include the following step 1001.

Step 1001: The electronic device inputs original information into an image generation model and outputs a derived image, or inputs original information into an image generation model, performs image processing, and outputs a derived image.

In some embodiments of the present application, the above-mentioned original information includes at least one of the following: an image to be processed, a description text, and a mask image.

In some embodiments of the present application, when the above-mentioned original information includes descriptive text, the electronic device inputs the descriptive text into the image generation model, and the image generation model performs a text-to-image task to generate and output a derivative image described by the descriptive text.

In some embodiments of the present application, when the above-mentioned original information includes descriptive text and an image to be processed, the electronic device inputs the descriptive text and the image to be processed into an image generation model, the image generation model executes the text-to-image generation task, performs image processing on the image to be processed according to the descriptive text, obtains a derivative image and outputs it.

For example, in the above text and image generation task of changing the style of the image to be processed, the above description text includes the modified image style. The electronic device inputs the description text and the image to be processed into the image generation model, and the image generation model modifies the style of the image to be processed according to the modified image style in the description text, obtains a derivative image and outputs it.

In some embodiments of the present application, when the above-mentioned original information includes descriptive text, an image to be processed and a mask image, the electronic device inputs the descriptive text, the image to be processed and the mask image into an image generation model, and the image generation model performs a local image modification task, modifies the image content of the mask area corresponding to the mask image in the image to be processed according to the descriptive text, obtains a derivative image and outputs it.

In some embodiments of the present application, when the above-mentioned original information includes an image to be processed and a mask image, the electronic device inputs the image to be processed and the mask image into an image generation model, and the image generation model performs an image elimination task, eliminates the image content of the mask area corresponding to the mask image in the image to be processed, obtains a derivative image and outputs it.

In some embodiments of the present application, when the above-mentioned original information includes descriptive text, an image to be processed and a mask image, the electronic device inputs the descriptive text, the image to be processed and the mask image into an image generation model, and the image generation model performs an image elimination task, eliminates the image content of the mask area corresponding to the mask image in the image to be processed, obtains a derivative image and outputs it.

In some embodiments of the present application, when the above-mentioned original information includes descriptive text, an image to be processed and a mask image, the electronic device inputs the descriptive text, the image to be processed and the mask image into an image generation model, the image generation model performs an image expansion task, predicts the image content of the mask area corresponding to the mask image in the image to be processed based on the descriptive text, obtains a derived image and outputs it.

In this way, the electronic device can process various types of image generation tasks using the trained image generation model, thereby improving the processing efficiency of the image generation tasks.

In some embodiments of the present application, when the original information includes an image to be processed, description text, and a mask image, the above step 1001 can be specifically implemented by the following steps 1101 to 1102.

Step 1101: The electronic device encodes the image to be processed through the first conditional encoder in the image generation model to obtain image feature information of the image to be processed, and encodes the description text through the second conditional encoder in the image generation model to obtain text feature information of the description text, and performs noise processing on the image to be processed based on the Gaussian noise matrix and the mask image through the noise addition module in the image generation model to obtain a noise image.

Step 1102: The electronic device performs image processing based on the image feature information, the text feature information and the noise image through the diffusion model in the image generation model to obtain a derivative image.

It should be noted that the specific implementation of the above steps 1101 to 1102 is the same as the implementation process of generating a derivative image in the above embodiment. The specific implementation can be found in the relevant description of the above embodiment, and this application will not repeat it here.

In this way, since the image generation model has learned the ability to handle various types of image generation tasks by controlling conditions during the model training process, the electronic device uses the trained image generation model to perform image processing on the image to be processed, when the image to be processed, descriptive text and mask image are input, to obtain a derivative image, thereby improving the processing efficiency of the image generation task.

In some embodiments of the present application, before the above step 1101, the training method of the image generation model provided by some embodiments of the present application also includes the following step 1201.

Step 1201: The electronic device performs feature compression on the image to be processed through the image encoder in the image generation model to obtain an image feature matrix.

In some embodiments of the present application, the above step 1101 can be specifically implemented through the following step 1301.

Step 1301: The electronic device performs noise processing on the image feature matrix based on the Gaussian noise matrix and the mask image through the noise addition module in the image generation model to obtain a noise image.

In some embodiments of the present application, after the above step 1102, the training method of the image generation model provided by some embodiments of the present application also includes the following step 1401.

Step 1401: The electronic device decodes the derived image through the image decoder in the image generation model and outputs the decoded derivative image.

It should be noted that the specific implementation of the above steps 1201, 1301 and 1401 is the same as the process of using the image encoder, noise addition module and image decoder in the above embodiments. The specific implementation can be found in the relevant description of the above embodiments, and this application will not repeat them here.

In this way, the electronic device uses an image encoder to perform feature compression on the image to be processed before performing noise addition processing, thereby reducing the amount of calculation in the noise addition process and the amount of data processing in the subsequent process of generating a derivative image, thereby improving the processing efficiency of the electronic device, and after obtaining the derivative image, decoding it through an image decoder to obtain a decoded derivative image with a higher resolution.

Exemplarily, when the original information includes descriptive text, the electronic device extracts text feature information of the descriptive text through the second conditional encoder in the image generation model, and then inputs the text feature information and the Gaussian noise matrix into the diffusion model in the image generation model for processing to obtain a derived image. The derived image is an image describing the text, thereby completing the text-generated image task.

Exemplarily, in the case where the original information includes a description text and an image to be processed, the description text includes image description information of the image to be processed and the modified image style of the image to be processed. The electronic device extracts image feature information of the image to be processed through a first conditional encoder in the image generation model, extracts text feature information of the description text through a second conditional encoder in the image generation model, performs feature compression on the image to be processed through an image encoder in the image generation model to obtain an image feature matrix, performs noise processing on the image feature matrix based on a Gaussian noise matrix through a noise addition module in the image generation model to obtain a noise image, inputs text feature information, image feature information and noise image into a diffusion model in the image generation model for processing to obtain a derivative image, and then decodes the derivative image through an image decoder to obtain a decoded derivative image, the decoded derivative image is an image obtained after style conversion of the image to be processed, and the text and image generation task is completed.

Exemplarily, in the case where the original information includes description text, an image to be processed and a mask image, the mask image is a mask image of a mask area in the image to be processed, and the description text includes image description information of the image to be processed. The electronic device extracts image feature information of the image to be processed through a first conditional encoder in the image generation model, extracts text feature information of the description text through a second conditional encoder in the image generation model, and performs feature compression on the image to be processed through an image encoder in the image generation model to obtain an image feature matrix. The electronic device performs noise processing on the image feature matrix based on a Gaussian noise matrix and a mask image through a noise adding module in the image generation model to obtain a noise image. The electronic device inputs the text feature information, the image feature information and the noise image into a diffusion model in the image generation model for processing to obtain a derivative image, and then decodes the derivative image through an image decoder to obtain a derivative image. The decoded derivative image is an image obtained after eliminating the image content of the mask area in the image to be processed, thereby completing the image elimination task.

Exemplarily, the original information includes description text, an image to be processed and a mask image, the mask image is a mask image of a mask area in the image to be processed, and the description text includes image description information of the image to be processed and the modified image content of the mask area. The electronic device extracts image feature information of the image to be processed through a first conditional encoder in the image generation model, extracts text feature information of the description text through a second conditional encoder in the image generation model, and performs feature compression on the image to be processed through an image encoder in the image generation model to obtain an image feature matrix. The electronic device performs noise processing on the image feature matrix based on a Gaussian noise matrix and a mask image through a noise adding module in the image generation model to obtain a noise image. The electronic device inputs the text feature information, the image feature information and the noise image into a diffusion model in the image generation model for processing to obtain a derivative image, and then decodes the derivative image through an image decoder to obtain a derivative image. The decoded derivative image is an image obtained by modifying the image content of the mask area in the image to be processed according to the description text, thereby completing the image local modification task.

Exemplarily, the original information includes description text, an image to be processed and a mask image, the mask image is a mask image of a mask area in the image to be processed, and the description text includes image description information of the image to be processed. The electronic device extracts image feature information of the image to be processed through a first conditional encoder in the image generation model, extracts text feature information of the description text through a second conditional encoder in the image generation model, and performs feature compression on the image to be processed through an image encoder in the image generation model to obtain an image feature matrix. The electronic device performs noise processing on the image feature matrix based on a Gaussian noise matrix and a mask image through a noise addition module in the image generation model to obtain a noise image. The electronic device inputs the text feature information, the image feature information and the noise image into a diffusion model in the image generation model for processing to obtain a derivative image, and then decodes the derivative image through an image decoder to obtain a derivative image. The decoded derivative image is an image obtained after predicting the image content of the mask area in the image to be processed according to the description text, thereby completing the image expansion task.

It should be noted that when the above-mentioned image generation model is used to perform text-to-image, text and image-to-image, image elimination, local image modification and image expansion tasks, the specific implementation process can be referred to the relevant description in the training method of the above-mentioned image generation model, and this application will not go into details here.

In some embodiments of the present application, when configuring an image generation model in an electronic device, a conditional encoder may be configured in the electronic device based on the computing power of the electronic device and the type of image generation task being performed.

For example, for electronic devices with limited computing power, a lightweight text encoder, i.e., a text encoder with a small number of parameters, can be configured in the conditional encoder. For electronic devices with greater computing power, only a heavyweight text encoder, i.e., a text encoder with a large number of parameters, can be configured in the conditional encoder, or a combination of a lightweight text encoder and a heavyweight text encoder can be configured in the conditional encoder to obtain better text feature information.

For example, a lightweight text encoder can be configured for mobile communication devices, vehicle equipment, etc. A lightweight text encoder and a heavyweight text encoder can be configured for a cloud server.

Exemplarily, if the electronic device is configured with an image generation model to process only text-based image tasks, since there is no image in the input, the image encoder may not be configured in the conditional encoder, thereby saving storage space of the electronic device.

Some embodiments of the present application provide a framework of a multifunctional image generation model, and in conjunction with the image generation model training method proposed in some embodiments of the present application, when an electronic device uses a training sample group for model training, the image generation model is controlled by controlling conditions to perform different types of image generation tasks, so that the processing of image generation tasks such as text-generated images, image-generated images, image elimination, local image modification, and image expansion are integrated into one image generation model. There is no need to train multiple models, which reduces the amount of data processed by the electronic device, thereby reducing the time consumed and the resources occupied, reducing the waste of resources of the electronic device, improving the efficiency of model training, and effectively reducing the development and deployment costs.

The training method of the image generation model provided in some embodiments of the present application may specifically include the following steps 2501-2517.

Step 2501: The electronic device obtains at least two training sample groups, each training sample group including a sample image, description text of each sample image, and a mask image.

Step 2502: The electronic device performs feature compression on at least one sample image through a convolution module in a VAE encoder to obtain at least one compressed image feature matrix.

In some embodiments of the present application, the number of pixels in each compressed image feature matrix is greater than the number of pixels in each image feature matrix.

Step 2503: The electronic device uses the self-attention module in the VAE encoder to extract features from each compressed image feature matrix based on the correlation between the pixels in each compressed image feature matrix to obtain at least one image feature matrix.

Step 2504: The electronic device performs weighted summation of the Gaussian noise matrix and the mask image to obtain a noisy matrix.

Step 2505: The electronic device performs weighted summation of the noise matrix and the sample image to obtain a noise image.

Step 2506: The electronic device divides each sample image into N sample image blocks through the image segmentation module in the first conditional encoder, and rearranges the pixels of each sample image block into a one-dimensional vector to obtain N one-dimensional vectors for each sample image, and splices the N one-dimensional vectors for each sample image to obtain a pixel rearranged image of each sample image.

Wherein, N is an integer greater than 1.

Step 2507: The electronic device extracts feature information of each pixel in each pixel rearrangement image through the mapping layer in the first conditional encoder to obtain at least one pixel rearrangement feature vector.

Step 2508: The electronic device uses the self-attention module in the first conditional encoder to extract features from each pixel rearrangement feature vector based on the correlation between pixels in each pixel rearrangement image, so as to obtain image feature information of each sample image.

It should be noted that the above steps 2506 to 2508 can be performed before step 2502 and after step 2501, or can be performed after step 2505, and this application does not limit this.

Step 2509: The electronic device divides each description text into at least two words through the word segmenter in the second conditional encoder, and encodes each word segment to obtain a text feature vector of each description text.

Step 2510: The electronic device uses the self-attention module in the second conditional encoder to extract features from the text feature vector of each description text based on the association between the word segments in each description text, so as to obtain text feature information of each description text.

It should be noted that the above steps 2509 to 2510 may be performed before step 2502 and after step 2501, or may be performed after step 2505, and this embodiment does not limit this.

In addition, there is no sequential execution order between the above steps 2506-2508 and steps 2509-2510.

Step 2511: The electronic device determines at least one item of conditional feature information based on the control condition, the at least one item of image feature information, and the at least one item of text feature information.

Step 2512: The electronic device divides each noise image into M noise image blocks through the image segmentation module in the diffusion model.

Wherein, M is an integer greater than 1;

Step 2513: The electronic device extracts feature information of each noise image block through a linear transformation operator in the diffusion model, and adds a position encoding vector to obtain feature information of each noise image.

Step 2514: The electronic device performs linear transformation processing on the at least one item of conditional feature information through the linear transformation operator in the diffusion model to obtain at least one item of conditional feature information after linear transformation.

Step 2515: The electronic device splices the at least one linearly transformed conditional feature information and the feature information of the at least one noise image through the splicing operator in the diffusion model to obtain at least one fused feature information.

Step 2516: The electronic device performs feature extraction on the at least one noisy vector through the diffusion self-attention module in the diffusion model to obtain at least one set of scaling factors, and performs feature scaling on the at least one fused feature information based on the at least one set of scaling factors to obtain at least one scaled feature information.

Each item of scaling feature information corresponds to a set of scaling factors and an item of fusion feature information.

Step 2517: The electronic device normalizes the at least one item of scaling feature information using a layer normalization operator in the diffusion model to obtain at least one item of noise feature information.

Step 2518: The electronic device performs channel reduction processing on the at least one item of noise feature information through the convolution operator in the diffusion model to obtain a noise matrix of the at least one noise image.

Step 2519: The electronic device determines at least one first loss value based on the Gaussian noise matrix and the noise matrix of the at least one noise image, and determines at least one second loss value based on the mask image of the at least one sample image and the noise matrix of the at least one noise image.

Step 2520: The electronic device performs weighted summation of the at least one first loss value and the at least one second loss value to obtain a noise loss value of the at least one sample image.

Step 2521: The electronic device adjusts the model parameters of the initial model based on the noise loss value of the at least one sample image to obtain an image generation model.

Step 2522: The electronic device obtains the image to be processed, the description text of the image to be processed, and the mask image.

In some embodiments of the present application, the mask image is a mask image of a mask area in the image to be processed, and the description text includes image description information of the image to be processed and image content after modifying the image content of the mask area.

Step 2523: The electronic device encodes the image to be processed through the first conditional encoder in the image generation model to obtain image feature information of the image to be processed, and encodes the description text through the second conditional encoder in the image generation model to obtain text feature information of the description text.

Step 2524: The electronic device performs feature compression on the image to be processed through the image encoder in the image generation model to obtain an image feature matrix.

Step 2525: The electronic device performs noise processing on the image feature matrix based on the Gaussian noise matrix and the mask image through the noise adding module in the image generation model to obtain the noise image.

The electronic device performs image processing based on the image feature information, the text feature information and the noise image through the diffusion model in the image generation model to obtain a derivative image.

Step 2526: The electronic device decodes the derivative image through an image decoder to obtain a decoded derivative image.

In some embodiments of the present application, the decoded derivative image is an image obtained by modifying the image content of the mask area in the image to be processed.

It should be noted that the above steps 2522 to 2526 are used to illustrate the use of the image generation model by taking the implementation of local image modification as an example, and do not limit the use of the image generation model.

It should be noted that the implementation process of the above steps 2501 to 2526 is the same as the implementation process in the embodiment of the training method of the above-mentioned image generation model. For the parts not described in detail in this embodiment, please refer to the relevant description of the above embodiment, and this application will not repeat them here.

It should be noted that the above-mentioned method embodiments, or various possible implementation methods in each method embodiment, can be executed separately, or any two or more can be executed in combination with each other. The specific implementation method can be determined according to actual usage requirements, and some embodiments of the present application do not limit this.

or,

It should be noted that the above-mentioned method embodiments, or various possible implementation methods in each method embodiment, can be executed separately, or, under the premise that there is no contradiction, can also be executed in combination with each other. The specific implementation can be determined according to actual usage requirements, and some embodiments of the present application do not limit this.

In some embodiments of the present application, the training method of the image generation model provided by the training device of the image generation model may be executed by the training device of the image generation model. In some embodiments of the present application, the training device of the image generation model provided by some embodiments of the present application is illustrated by taking the training method of the image generation model performed by the training device of the image generation model as an example.

FIG18 is a schematic diagram of the structure of a training device for an image generation model provided in some embodiments of the present application, the device comprising:

An acquisition unit 1701 is used to acquire at least two training sample groups, each training sample group includes a sample image, description text of each sample image, and a mask image;

The encoding unit 1702 is used to encode at least one sample image in the at least two training sample groups acquired by the acquisition unit to obtain at least one image feature information, and to encode the description text of each sample image to obtain at least one text feature information;

A determining unit 1703 is used to determine at least one conditional feature information based on the control condition, at least one image feature information and at least one text feature information obtained by the encoding unit; the control condition is used to indicate the type of the image generation task, and the conditional feature information is determined according to the type of the image generation task;

The training unit 1704 is used to train the initial model based on at least one sample image acquired by the acquisition unit, the mask image of at least one sample image and at least one conditional feature information determined by the determination unit to obtain an image generation model.

In some embodiments of the present application, the determining unit 1703 is specifically configured to:

In a case where the type of the image generation task indicated by the control condition is a text image task, determining at least one item of text feature information as at least one item of condition feature information;

In a case where the type of the image generation task indicated by the control condition is an image processing task, determining at least one item of image feature information and at least one item of text feature information as at least one item of conditional feature information;

Among them, the image processing task includes at least one of the following: an image-to-image task, a text-to-image task, an image elimination task, an image local modification task, and an image expansion task.

In some embodiments of the present application, the device further includes:

a processing unit, configured to perform noise addition processing on at least one sample image based on the Gaussian noise matrix and the mask image of at least one sample image acquired by the acquisition unit, so as to obtain at least one noise image;

A training unit, configured to train the initial model based on at least one condition feature information determined by the determination unit and at least one noise image processed by the processing unit to obtain an image generation model;

Among them, each noise image corresponds to a conditional feature information.

In some embodiments of the present application, the processing unit is specifically used to:

Performing a weighted summation of the Gaussian noise matrix and the mask image of at least one sample image acquired by the acquisition unit to obtain at least one noise matrix; and performing a weighted summation of the at least one noise matrix and at least one sample image acquired by the acquisition unit to obtain at least one noise image;

Among them, each noisy image corresponds to a noise matrix and a sample image.

In some embodiments of the present application, the device further includes:

A compression unit, configured to perform feature compression on at least one sample image acquired by the acquisition unit through an image encoder to obtain at least one image feature matrix;

A processing unit, configured to perform noise addition processing on at least one image feature matrix based on the Gaussian noise matrix and the mask image of at least one sample image acquired by the acquisition unit, so as to obtain at least one noise image;

The training unit is used to train the initial model based on at least one condition feature information determined by the determination unit and at least one noise image acquired by the acquisition unit to obtain an image generation model.

In some embodiments of the present application, the compression unit is specifically used for:

Performing feature compression on at least one sample image acquired by the acquisition unit through a convolution module in the image encoder to obtain at least one compressed image feature matrix, wherein the number of pixel points in each compressed image feature matrix is greater than the number of pixel points in each image feature matrix;

Through the self-attention module in the image encoder, feature extraction is performed on each compressed image feature matrix based on the correlation relationship between the pixels in each compressed image feature matrix to obtain at least one image feature matrix.

In some embodiments of the present application, the acquisition unit 1701 is further configured to:

Acquire at least two sample images, image description information of each sample image, and annotation information of at least two mask areas of each sample image, wherein the at least two mask areas of each sample image are obtained by predicting the mask areas of each sample image using the segmentation model and segmenting each sample image based on the prediction results, and the annotation information of each mask area includes at least one of the following: the area of each mask area, the prediction accuracy of each mask area, and the prediction stability of each mask area; the prediction accuracy represents the prediction accuracy of the mask area using the segmentation model, and the prediction stability represents the prediction stability of the mask area using the segmentation model;

Based on the annotation information of at least two mask areas of each sample image, determining a first mask area whose annotation information satisfies a filtering condition from the at least two mask areas of each sample image, where the filtering condition includes at least one of the following: an area of the mask area is within a preset range, a prediction accuracy of the mask area is greater than an accuracy threshold, and a prediction stability of the mask area is greater than a stability threshold;

Adding the description information of the first mask area of each sample image to the image description information of each sample image to obtain a description text of each sample image;

Each sample image, the description text of each sample image and the mask image of each sample image are constructed into a training sample group to obtain at least two training sample groups, wherein the mask image of each sample image is the mask image corresponding to the first mask area in each sample image.

In some embodiments of the present application, the acquisition unit 1701 is further configured to:

Obtain at least two sample images and description text for each sample image;

generating a second mask region in each sample image according to the size range and size information of each sample image;

Each sample image, the description text of each sample image, and the mask image of each sample image are constructed into a training sample group to obtain at least two training sample groups, and the mask image of each sample image is the mask image of the second mask area in each sample image.

In some embodiments of the present application, the encoding unit 1702 is specifically used to:

By using the image segmentation module in the first conditional encoder, each sample image acquired by the acquisition unit is segmented into N sample image blocks, and the pixels of each sample image block are rearranged into a one-dimensional vector to obtain N one-dimensional vectors of each sample image, and the N one-dimensional vectors of each sample image are spliced to obtain a pixel rearranged image of each sample image, where N is an integer greater than 1;

Extracting feature information of each pixel in each pixel rearrangement image through a mapping layer in the first conditional encoder to obtain at least one pixel rearrangement feature vector;

Through the self-attention module in the first conditional encoder, based on the correlation between pixels in each pixel rearrangement image, feature extraction is performed on each pixel rearrangement feature vector to obtain image feature information of each sample image.

In some embodiments of the present application, the encoding unit 1702 is specifically used to:

Each description text acquired by the acquisition unit is divided into at least two words by a word segmenter in the second conditional encoder, and each word is encoded to obtain a text feature vector of each description text;

Through the self-attention module in the second conditional encoder, based on the association relationship between the segmented words in each description text, feature extraction is performed on the text feature vector of each description text to obtain the text feature information of each description text.

In some embodiments of the present application, the training unit 1704 is specifically used to:

Acquire feature information of at least one noise image through a diffusion model, and fuse at least one condition feature information and at least one noise image feature information to obtain at least one fused feature information, where each fused feature information corresponds to one condition feature information and one noise image feature information;

Determining a noise matrix of at least one noisy image based on at least one fused feature information and at least one noisy vector by a diffusion model, each noisy vector representing a noise level of a noisy image;

Determine a noise loss value of at least one sample image based on a noise matrix of at least one noise image, a Gaussian noise matrix, and a mask image of at least one sample image acquired by an acquisition unit;

Based on the noise loss value of at least one sample image, the model parameters of the initial model are adjusted to obtain an image generation model.

In some embodiments of the present application, the device further includes:

The extraction unit is used to obtain feature information of at least one noise image through a diffusion model, and fuse at least one conditional feature information and feature information of at least one noise image to obtain at least one fused feature information, divide each noise image into M noise image blocks through an image segmentation module in the diffusion model, where M is an integer greater than 1; and extract feature information of each noise image block through a linear transformation operator in the diffusion model, and add a position encoding vector to obtain feature information of each noise image.

In some embodiments of the present application, the training unit 1704 is specifically used to:

Performing linear transformation processing on at least one item of conditional feature information through a linear transformation operator in a diffusion model to obtain at least one item of conditional feature information after linear transformation;

At least one item of conditional feature information after linear transformation and at least one item of feature information of a noise image are spliced by a splicing operator in a diffusion model to obtain at least one item of fused feature information.

In some embodiments of the present application, the training unit 1704 is specifically used to:

Performing feature extraction on at least one noisy vector through a diffusion self-attention module in a diffusion model to obtain at least one set of scaling factors, and performing feature scaling on at least one fused feature information based on the at least one set of scaling factors to obtain at least one scaled feature information, wherein each scaled feature information corresponds to a set of scaling factors and one fused feature information;

Normalizing at least one item of scaling feature information by a layer normalization operator in a diffusion model to obtain at least one item of noise feature information;

Through the convolution operator in the diffusion model, at least one noise feature information is subjected to channel reduction processing to obtain a noise matrix of at least one noise image.

In some embodiments of the present application, the training unit 1704 is specifically used to:

Determining at least one first loss value based on the Gaussian noise matrix and the noise matrix of the at least one noise image, and determining at least one second loss value based on the mask image of the at least one sample image and the noise matrix of the at least one noise image;

The at least one first loss value and the at least one second loss value are weightedly summed to obtain a noise loss value of at least one sample image.

In some embodiments of the present application, the device further includes:

an execution unit, configured to, after training the initial model based on at least one sample image acquired by the acquisition unit, the mask image of at least one sample image, and at least one conditional feature information determined by the determination unit to obtain an image generation model, input the original information into the image generation model and output a derived image, or input the original information into the image generation model, perform image processing, and output a derived image;

The original information includes at least one of the following: an image to be processed, a description text, and a mask image.

In some embodiments of the present application, the execution unit is specifically used to, when the original information includes an image to be processed, a description text and a mask image, encode the image to be processed through a first conditional encoder in an image generation model to obtain image feature information of the image to be processed, and encode the description text through a second conditional encoder in the image generation model to obtain text feature information of the description text, and perform noise processing on the image to be processed based on a Gaussian noise matrix and a mask image through a noise addition module in the image generation model to obtain a noise image; and perform image processing based on the image feature information, text feature information and the noise image through a diffusion model in the image generation model to obtain a derivative image.

In some embodiments of the present application, the device further includes:

A compression unit is used for performing feature compression on the image to be processed based on the Gaussian noise matrix and the mask image through the noise adding module in the image generation model to obtain the image feature matrix before obtaining the noise image;

A processing unit, configured to perform noise processing on the image feature matrix based on the Gaussian noise matrix and the mask image through a noise adding module in the image generation model to obtain a noise image;

The decoding unit is used to decode the derivative image obtained by the execution unit through the image decoder in the image generation model, and output the decoded derivative image.

Some embodiments of the present application provide a training device for an image generation model. When an electronic device uses a training sample group for model training, the image generation model is controlled by controlling conditions to perform different types of image generation tasks, so that the processing of image generation tasks such as text-generated images, image-generated images, image elimination, local image modification, and image expansion are integrated into one image generation model. There is no need to train multiple models, which reduces the amount of data processed by the electronic device, thereby reducing the time consumed and the resources occupied, reducing the waste of resources of the electronic device, improving the efficiency of model training, and effectively reducing the development and deployment costs.

In some embodiments of the present application, the training device of the image generation model can be an electronic device or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR)/virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and some embodiments of the present application are not specifically limited.

In some embodiments of the present application, the training device of the image generation model may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in some embodiments of the present application.

The image generation model training device provided in some embodiments of the present application is capable of implementing the various processes implemented in the various embodiments of the above-mentioned image generation model training method. To avoid repetition, they will not be described again here.

Optionally, as shown in Figure 19, some embodiments of the present application also provide an electronic device 2700, including a processor 2701 and a memory 2702, and the memory 2702 stores programs or instructions that can be executed on the processor 2701. When the program or instructions are executed by the processor 2701, the various steps of the various embodiments of the training method of the above-mentioned image generation model are implemented, and the same technical effect can be achieved. To avoid repetition, they are not repeated here.

It should be noted that the electronic devices in some embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

FIG20 is a schematic diagram of the hardware structure of an electronic device implementing some embodiments of the present application.

The electronic device 2800 includes but is not limited to: a radio frequency unit 2801, a network module 2802, an audio output unit 2803, an input unit 2804, a sensor 2805, a display unit 2806, a user input unit 2807, an interface unit 2808, a memory 2809, and a processor 2810 and other components.

Those skilled in the art will appreciate that the electronic device 2800 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 2810 through a power management system, so that the power management system can manage charging, discharging, and power consumption management. The electronic device structure shown in FIG20 does not constitute a limitation on the electronic device, and the electronic device may include more or fewer components than shown, or combine certain components, or arrange components differently, which will not be described in detail here.

The processor 2810 is used for:

Obtain at least two training sample groups, each training sample group including a sample image, description text of each sample image, and a mask image;

Encoding at least one sample image in at least two training sample groups to obtain at least one item of image feature information, and encoding a description text of each sample image to obtain at least one item of text feature information;

Determining at least one conditional feature information based on a control condition, at least one image feature information, and at least one text feature information; the control condition is used to indicate the type of the image generation task, and the conditional feature information is determined according to the type of the image generation task;

Based on at least one sample image, at least one mask image of the sample image and at least one conditional feature information, an initial model is trained to obtain an image generation model.

Optionally, the processor 2810 is specifically configured to, when the type of the image generation task indicated by the control condition is a text image task, determine at least one item of text feature information as at least one item of condition feature information; when the type of the image generation task indicated by the control condition is an image processing task, determine at least one item of image feature information and at least one item of text feature information as at least one item of condition feature information;

Among them, the image processing task includes at least one of the following: an image-to-image task, a text-to-image task, an image elimination task, an image local modification task, and an image expansion task.

Optionally, the processor 2810 is specifically configured to:

Based on the Gaussian noise matrix and the mask image of the at least one sample image, performing noise processing on the at least one sample image to obtain at least one noise image;

Based on at least one conditional feature information and at least one noise image, an initial model is trained to obtain an image generation model;

Among them, each noise image corresponds to a conditional feature information.

Optionally, the processor 2810 is specifically configured to:

Performing weighted summation on the Gaussian noise matrix and the mask image of at least one sample image to obtain at least one noise matrix;

Performing weighted summation on at least one noise matrix and at least one sample image to obtain at least one noise image;

Among them, each noisy image corresponds to a noise matrix and a sample image.

Optionally, the processor 2810 is specifically configured to:

Performing feature compression on at least one sample image by using an image encoder to obtain at least one image feature matrix;

Based on the Gaussian noise matrix and the mask image of at least one sample image, performing noise processing on at least one image feature matrix to obtain at least one noise image;

Based on at least one conditional feature information and at least one noise image, the initial model is trained to obtain an image generation model.

Optionally, the processor 2810 is specifically configured to:

Performing feature compression on at least one sample image through a convolution module in an image encoder to obtain at least one compressed image feature matrix, wherein the number of pixel points in each compressed image feature matrix is greater than the number of pixel points in each image feature matrix;

Through the self-attention module in the image encoder, feature extraction is performed on each compressed image feature matrix based on the correlation relationship between the pixels in each compressed image feature matrix to obtain at least one image feature matrix.

Optionally, before acquiring at least two training sample groups, the processor 2810 is further configured to:

Acquire at least two sample images, image description information of each sample image, and annotation information of at least two mask areas of each sample image, wherein the at least two mask areas of each sample image are obtained by predicting the mask areas of each sample image using the segmentation model and segmenting each sample image based on the prediction results, and the annotation information of each mask area includes at least one of the following: the area of each mask area, the prediction accuracy of each mask area, and the prediction stability of each mask area; the prediction accuracy represents the prediction accuracy of the mask area using the segmentation model, and the prediction stability represents the prediction stability of the mask area using the segmentation model;

Based on the annotation information of at least two mask areas of each sample image, determining a first mask area whose annotation information satisfies a filtering condition from the at least two mask areas of each sample image, where the filtering condition includes at least one of the following: an area of the mask area is within a preset range, a prediction accuracy of the mask area is greater than an accuracy threshold, and a prediction stability of the mask area is greater than a stability threshold;

Adding the description information of the first mask area of each sample image to the image description information of each sample image to obtain a description text of each sample image;

Each sample image, the description text of each sample image and the mask image of each sample image are constructed into a training sample group to obtain at least two training sample groups, wherein the mask image of each sample image is the mask image corresponding to the first mask area in each sample image.

Optionally, before acquiring at least two training sample groups, the processor 2810 is further configured to:

Obtain at least two sample images and description text for each sample image;

generating a second mask region in each sample image according to the size range and size information of each sample image;

Each sample image, the description text of each sample image, and the mask image of each sample image are constructed into a training sample group to obtain at least two training sample groups, and the mask image of each sample image is the mask image of the second mask area in each sample image.

Optionally, the processor 2810 is specifically configured to:

By using the image segmentation module in the first conditional encoder, each sample image is segmented into N sample image blocks, and the pixels of each sample image block are rearranged into a one-dimensional vector to obtain N one-dimensional vectors of each sample image, and the N one-dimensional vectors of each sample image are spliced to obtain a pixel rearranged image of each sample image, where N is an integer greater than 1;

Extracting feature information of each pixel in each pixel rearrangement image through a mapping layer in the first conditional encoder to obtain at least one pixel rearrangement feature vector;

Through the self-attention module in the first conditional encoder, based on the correlation between pixels in each pixel rearrangement image, feature extraction is performed on each pixel rearrangement feature vector to obtain image feature information of each sample image.

Optionally, the processor 2810 is specifically configured to:

Each description text is divided into at least two words by a word segmenter in the second conditional encoder, and each word is encoded to obtain a text feature vector of each description text;

Through the self-attention module in the second conditional encoder, based on the association relationship between the segmented words in each description text, feature extraction is performed on the text feature vector of each description text to obtain the text feature information of each description text.

Optionally, the processor 2810 is specifically configured to:

Acquire feature information of at least one noise image through a diffusion model, and fuse at least one condition feature information and at least one noise image feature information to obtain at least one fused feature information, where each fused feature information corresponds to one condition feature information and one noise image feature information;

Determining a noise matrix of at least one noisy image based on at least one fused feature information and at least one noisy vector by a diffusion model, each noisy vector representing a noise level of a noisy image;

Determining a noise loss value of at least one sample image based on a noise matrix of at least one noise image, a Gaussian noise matrix, and a mask image of at least one sample image;

Based on the noise loss value of at least one sample image, the model parameters of the initial model are adjusted to obtain an image generation model.

Optionally, before acquiring feature information of at least one noise image through a diffusion model and fusing at least one condition feature information with feature information of at least one noise image to obtain at least one fused feature information, the processor 2810 is further configured to:

Each noise image is segmented into M noise image blocks through the image segmentation module in the diffusion model, where M is an integer greater than 1;

The feature information of each noise image block is extracted through the linear transformation operator in the diffusion model, and the position encoding vector is added to obtain the feature information of each noise image.

Optionally, the processor 2810 is specifically configured to:

Performing linear transformation processing on at least one item of conditional feature information through a linear transformation operator in a diffusion model to obtain at least one item of conditional feature information after linear transformation;

At least one item of conditional feature information after linear transformation and at least one item of feature information of a noise image are spliced by a splicing operator in a diffusion model to obtain at least one item of fused feature information.

Optionally, the processor 2810 is specifically configured to:

Performing feature extraction on at least one noisy vector through a diffusion self-attention module in a diffusion model to obtain at least one set of scaling factors, and performing feature scaling on at least one fused feature information based on the at least one set of scaling factors to obtain at least one scaled feature information, wherein each scaled feature information corresponds to a set of scaling factors and one fused feature information;

Normalizing at least one item of scaling feature information by a layer normalization operator in a diffusion model to obtain at least one item of noise feature information;

Through the convolution operator in the diffusion model, at least one noise feature information is subjected to channel reduction processing to obtain a noise matrix of at least one noise image.

Optionally, the processor 2810 is specifically configured to:

Determining at least one first loss value based on the Gaussian noise matrix and the noise matrix of the at least one noise image, and determining at least one second loss value based on the mask image of the at least one sample image and the noise matrix of the at least one noise image;

The at least one first loss value and the at least one second loss value are weightedly summed to obtain a noise loss value of at least one sample image.

Optionally, after the initial model is trained based on at least one sample image, at least one mask image of the sample image and at least one conditional feature information to obtain the image generation model, the processor 2810 is further configured to:

Inputting original information into an image generation model and outputting a derived image, or inputting original information into an image generation model, performing image processing, and outputting a derived image;

The original information includes at least one of the following: an image to be processed, a description text, and a mask image.

Optionally, the processor 2810 is specifically configured to:

The image to be processed is encoded by the first conditional encoder in the image generation model to obtain image feature information of the image to be processed, and the description text is encoded by the second conditional encoder in the image generation model to obtain text feature information of the description text, and the image to be processed is subjected to noise processing based on the Gaussian noise matrix and the mask image by the noise addition module in the image generation model to obtain a noise image;

Through the diffusion model in the image generation model, image processing is performed based on image feature information, text feature information and noise image to obtain a derived image.

Optionally, before performing noise processing on the image to be processed based on the Gaussian noise matrix and the mask image through the noise adding module in the image generation model to obtain the noisy image, the processor 2810 is further used to:

Through the image encoder in the image generation model, the image to be processed is feature compressed to obtain an image feature matrix;

Optionally, the processor 2810 is specifically configured to:

Through the noise adding module in the image generation model, based on the Gaussian noise matrix and the mask image, the image feature matrix is subjected to noise adding processing to obtain a noise image;

Optionally, after performing image processing based on the image feature information, the text feature information and the noise image through the diffusion model in the image generation model to obtain a derived image, the processor 2810 is further configured to:

The derived image is decoded by the image decoder in the image generation model, and the decoded derivative image is output.

Some embodiments of the present application provide a training method for an image generation model. When an electronic device uses a training sample group for model training, the image generation model is controlled by controlling conditions to perform different types of image generation tasks, so that the processing of image generation tasks such as text-generated images, image-generated images, image elimination, local image modification, and image expansion are integrated into one image generation model. There is no need to train multiple models, which reduces the amount of data processed by the electronic device, thereby reducing the time consumed and the resources occupied, reducing the waste of resources of the electronic device, improving the efficiency of model training, and effectively reducing the development and deployment costs.

It should be understood that in some embodiments of the present application, the input unit 2804 may include a graphics processor (GPU) 28041 and a microphone 28042, and the graphics processor 28041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 2806 may include a display panel 28061, and the display panel 28061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 2807 includes a touch panel 28071 and at least one of other input devices 28072. The touch panel 28071 is also called a touch screen. The touch panel 28071 may include two parts: a touch detection device and a touch controller. Other input devices 28072 may include, but are not limited to, a physical keyboard, function keys (such as a volume control button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

The memory 2809 can be used to store software programs and various data. The memory 2809 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 2809 may include a volatile memory or a non-volatile memory, or the memory 2809 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 2809 in some embodiments of the present application includes but is not limited to these and any other suitable types of memory.

The processor 2810 may include one or more processing units; optionally, the processor 2810 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 2810.

Some embodiments of the present application also provide a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the training method embodiment of the above-mentioned image generation model are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

Some embodiments of the present application further provide a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image generation model training method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

It should be understood that the chip mentioned in some embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

Some embodiments of the present application provide a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the training method embodiment of the image generation model as described above, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises one..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be noted that the scope of the method and device in the embodiment of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM/RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims (40)

1. An image generation model, characterized in that it comprises: a conditional encoder, a noise adding module and a diffusion model, wherein the conditional encoder and the noise adding module are respectively connected to the diffusion model; The conditional encoder is used to encode at least one original image to obtain at least one image feature information, and to encode the description text of each original image to obtain at least one text feature information; and to determine at least one conditional feature information based on a control condition, the at least one image feature information and the at least one text feature information; the control condition is used to indicate a type of an image generation task, and the conditional feature information is determined according to the type of the image generation task; The noise adding module is used to perform noise adding processing on the at least one original image based on the Gaussian noise matrix and the mask image of the at least one original image to obtain at least one noisy image; The diffusion model is used to generate at least one derivative image based on the at least one conditional feature information and the at least one noise image, and each derivative image corresponds to one conditional feature information and one noise image. 2. The model according to claim 1, characterized in that the conditional encoder is specifically used for: In a case where the type of the image generation task is a text image task, determining the at least one item of text feature information as the at least one item of conditional feature information; When the type of the image generation task is an image processing task, the at least one item of image feature information and the at least one item of text feature information are determined as the at least one item of conditional feature information; wherein the image processing task includes at least one of the following: an image-to-image task, a text-to-image task, an image elimination task, an image local modification task, and an image expansion task. 3. The model according to claim 1 or 2, characterized in that the image generation model further comprises: an image encoder and an image decoder, the image encoder is connected to the noise addition module, and the image decoder is connected to the diffusion model; The image encoder is used to perform feature compression on the at least one original image to obtain at least one image feature matrix; The noise adding module is specifically used to perform noise adding processing on the at least one image feature matrix based on the Gaussian noise matrix and the mask image of the at least one original image to obtain at least one noise image; The image decoder is used to decode the at least one derived image. 4. A training method for an image generation model, comprising: Obtain at least two training sample groups, each training sample group including a sample image, description text of each sample image, and a mask image; Encoding at least one sample image in the at least two training sample groups to obtain at least one item of image feature information, and encoding a description text of each sample image to obtain at least one item of text feature information; Determine at least one item of conditional feature information based on a control condition, the at least one item of image feature information and the at least one item of text feature information; the control condition is used to indicate a type of an image generation task, and the conditional feature information is determined according to the type of the image generation task; Based on the at least one sample image, the mask image of the at least one sample image and the at least one conditional feature information, the initial model is trained to obtain an image generation model. 5. The method according to claim 4, characterized in that the determining at least one conditional feature information based on the control condition, the at least one image feature information and the at least one text feature information comprises: In a case where the type of the image generation task indicated by the control condition is a text image task, determining the at least one item of text feature information as the at least one item of condition feature information; In a case where the type of the image generation task indicated by the control condition is an image processing task, determining the at least one item of image feature information and the at least one item of text feature information as the at least one item of conditional feature information; The image processing task includes at least one of the following: an image-to-image task, a text-to-image task, an image elimination task, an image local modification task, and an image expansion task. 6. The method according to claim 4, characterized in that the training of the initial model based on the at least one sample image, the mask image of the at least one sample image and the at least one conditional feature information to obtain the image generation model comprises: Based on the Gaussian noise matrix and the mask image of the at least one sample image, performing noise processing on the at least one sample image to obtain at least one noise image; Based on the at least one condition feature information and the at least one noise image, training the initial model to obtain the image generation model; Among them, each noise image corresponds to a conditional feature information. 7. The method according to claim 6, characterized in that the step of performing noise processing on the at least one sample image based on the Gaussian noise matrix and the mask image of the at least one sample image to obtain the at least one noise image comprises: Performing weighted summation on the Gaussian noise matrix and the mask image of the at least one sample image to obtain at least one noise matrix; Performing weighted summation on the at least one noise matrix and the at least one sample image to obtain at least one noise image; Among them, each noisy image corresponds to a noise matrix and a sample image. 8. The method according to claim 4, characterized in that the training of the initial model based on the at least one sample image, the mask image of the at least one sample image and the at least one conditional feature information to obtain the image generation model comprises: Performing feature compression on the at least one sample image by an image encoder to obtain at least one image feature matrix; Based on the Gaussian noise matrix and the mask image of the at least one sample image, performing noise processing on the at least one image feature matrix to obtain at least one noise image; Based on the at least one conditional feature information and the at least one noise image, the initial model is trained to obtain an image generation model. 9. The method according to claim 8, characterized in that the step of performing feature compression on the at least one sample image by using an image encoder to obtain at least one image feature matrix comprises: Performing feature compression on the at least one sample image through a convolution module in the image encoder to obtain at least one compressed image feature matrix, wherein the number of pixel points in each compressed image feature matrix is greater than the number of pixel points in each image feature matrix; Through the self-attention module in the image encoder, feature extraction is performed on each compressed image feature matrix based on the correlation relationship between the pixels in each compressed image feature matrix to obtain at least one image feature matrix. 10. The method according to claim 4, characterized in that before obtaining at least two training sample groups, the method further comprises: Acquire at least two sample images, image description information of each sample image, and annotation information of at least two mask areas of each sample image, wherein the at least two mask areas of each sample image are obtained by predicting the mask areas of each sample image using a segmentation model and segmenting each sample image based on the prediction results, and the annotation information of each mask area includes at least one of the following: the area of each mask area, the prediction accuracy of each mask area, and the prediction stability of each mask area; the prediction accuracy represents the prediction accuracy of the mask area using the segmentation model, and the prediction stability represents the prediction stability of the mask area using the segmentation model; Based on the annotation information of at least two mask areas of each sample image, determining a first mask area whose annotation information satisfies a filtering condition from the at least two mask areas of each sample image, wherein the filtering condition includes at least one of the following: an area of the mask area is within a preset range, a prediction accuracy of the mask area is greater than an accuracy threshold, and a prediction stability of the mask area is greater than a stability threshold; Adding the description information of the first mask area of each sample image to the image description information of each sample image to obtain a description text of each sample image; Each sample image, the description text of each sample image and the mask image of each sample image are constructed into a training sample group to obtain at least two training sample groups, wherein the mask image of each sample image is the mask image corresponding to the first mask area in each sample image. 11. The method according to claim 4, characterized in that before obtaining at least two training sample groups, the method further comprises: Obtain at least two sample images and description text for each sample image; generating a second mask region in each sample image according to the size range and size information of each sample image; Each sample image, the description text of each sample image, and the mask image of each sample image are constructed into a training sample group to obtain the at least two training sample groups, wherein the mask image of each sample image is the mask image of the second mask area in each sample image. 12. The method according to claim 4, characterized in that encoding at least one sample image in the at least two training sample groups to obtain at least one item of image feature information comprises: By using the image segmentation module in the first conditional encoder, each sample image is segmented into N sample image blocks, and the pixels of each sample image block are rearranged into a one-dimensional vector to obtain N one-dimensional vectors of each sample image, and the N one-dimensional vectors of each sample image are spliced to obtain a pixel rearranged image of each sample image, where N is an integer greater than 1; Extracting feature information of each pixel in each pixel rearrangement image through a mapping layer in the first conditional encoder to obtain at least one pixel rearrangement feature vector; Through the self-attention module in the first conditional encoder, based on the correlation relationship between pixels in each pixel rearrangement image, feature extraction is performed on each pixel rearrangement feature vector to obtain image feature information of each sample image. 13. The method according to claim 4, characterized in that the encoding of the description text of each sample image to obtain at least one text feature information comprises: Each description text is divided into at least two words by a word segmenter in the second conditional encoder, and each word is encoded to obtain a text feature vector of each description text; Through the self-attention module in the second conditional encoder, based on the association relationship between the segmented words in each description text, feature extraction is performed on the text feature vector of each description text to obtain text feature information of each description text. 14. The method according to claim 6 or 8, characterized in that the training of the initial model based on the at least one conditional feature information and the at least one noise image to obtain the image generation model comprises: Acquire feature information of the at least one noise image through a diffusion model, and fuse the at least one condition feature information with the feature information of the at least one noise image to obtain at least one fused feature information, where each fused feature information corresponds to one condition feature information and one piece of feature information of the noise image; Determining a noise matrix of the at least one noisy image based on the at least one fused feature information and at least one noise vector by means of the diffusion model, each noise vector representing a noise level of a noisy image; Determining a noise loss value of the at least one sample image based on a noise matrix of the at least one noise image, the Gaussian noise matrix, and a mask image of the at least one sample image; Based on the noise loss value of the at least one sample image, the model parameters of the initial model are adjusted to obtain an image generation model. 15. The method according to claim 14, characterized in that before obtaining the feature information of the at least one noise image through the diffusion model and fusing the at least one conditional feature information with the feature information of the at least one noise image to obtain at least one fused feature information, the method further comprises: Each noise image is divided into M noise image blocks by the image segmentation module in the diffusion model, where M is an integer greater than 1; The feature information of each noise image block is extracted by the linear transformation operator in the diffusion model, and the position encoding vector is added to obtain the feature information of each noise image. 16. The method according to claim 14, characterized in that the step of fusing the at least one conditional feature information and the feature information of the at least one noise image through a diffusion model to obtain at least one fused feature information comprises: Performing linear transformation processing on the at least one item of conditional feature information by using a linear transformation operator in the diffusion model to obtain at least one item of conditional feature information after linear transformation; The at least one linearly transformed conditional feature information and the feature information of the at least one noise image are spliced by a splicing operator in the diffusion model to obtain at least one fused feature information. 17. The method according to claim 14, characterized in that the step of determining the noise matrix of the at least one noise image based on the at least one fused feature information and the at least one noise vector through the diffusion model comprises: By using the diffusion self-attention module in the diffusion model, feature extraction is performed on the at least one noise-added vector to obtain at least one set of scaling factors, and based on the at least one set of scaling factors, feature scaling is performed on the at least one fused feature information to obtain at least one scaled feature information, where each scaled feature information corresponds to a set of scaling factors and one fused feature information; Normalizing the at least one item of scaling feature information by a layer normalization operator in the diffusion model to obtain at least one item of noise feature information; The at least one noise feature information is subjected to channel number reduction processing by using a convolution operator in the diffusion model to obtain a noise matrix of the at least one noise image. 18. The method according to claim 14, characterized in that the determining the noise loss value of the at least one sample image based on the noise matrix of the at least one noise image, the Gaussian noise matrix and the mask image of the at least one sample image comprises: Determining at least one first loss value based on the Gaussian noise matrix and the noise matrix of the at least one noise image, and determining at least one second loss value based on the mask image of the at least one sample image and the noise matrix of the at least one noise image; The at least one first loss value and the at least one second loss value are weightedly summed to obtain a noise loss value of the at least one sample image. 19. The method according to claim 4, characterized in that after the initial model is trained based on the at least one sample image, the mask image of the at least one sample image and the at least one conditional feature information to obtain the image generation model, the method further comprises: Inputting original information into the image generation model and outputting a derived image, or inputting original information into the image generation model, performing image processing, and outputting a derived image; The original information includes at least one of the following: an image to be processed, a description text, and a mask image. 20. The method according to claim 19, characterized in that, when the original information includes the image to be processed, the description text and the mask image, the inputting the original information into the image generation model, generating an image or performing image processing, and outputting a derived image comprises: The image to be processed is encoded by a first conditional encoder in the image generation model to obtain image feature information of the image to be processed, and the description text is encoded by a second conditional encoder in the image generation model to obtain text feature information of the description text, and the image to be processed is subjected to noise processing based on a Gaussian noise matrix and the mask image by a noise addition module in the image generation model to obtain a noise image; By using the diffusion model in the image generation model, image processing is performed based on the image feature information, the text feature information and the noise image to obtain a derivative image. 21. The method according to claim 20, characterized in that, before the image to be processed is subjected to noise processing based on a Gaussian noise matrix and the mask image by means of a noise adding module in the image generation model to obtain a noise image, the method further comprises: Performing feature compression on the image to be processed by an image encoder in the image generation model to obtain an image feature matrix; The step of performing noise processing on the image to be processed based on a Gaussian noise matrix and the mask image by using a noise adding module in the image generation model to obtain a noise image includes: By using the noise adding module in the image generation model, based on the Gaussian noise matrix and the mask image, the image feature matrix is subjected to noise adding processing to obtain the noise image; After performing image processing based on the image feature information, the text feature information and the noise image through the diffusion model in the image generation model to obtain a derived image, the method further includes: The derivative image is decoded by an image decoder in the image generation model, and a decoded derivative image is output. 22. A training device for an image generation model, characterized in that the device comprises: An acquisition unit, used for acquiring at least two training sample groups, each training sample group including a sample image, description text of each sample image and a mask image; an encoding unit, configured to encode at least one sample image in the at least two training sample groups acquired by the acquisition unit to obtain at least one item of image feature information, and to encode a description text of each sample image to obtain at least one item of text feature information; a determining unit, configured to determine at least one conditional feature information based on a control condition, the at least one image feature information obtained by the encoding unit, and the at least one text feature information; the control condition is used to indicate a type of an image generation task, and the conditional feature information is determined according to the type of the image generation task; A training unit is used to train the initial model based on the at least one sample image acquired by the acquisition unit, the mask image of the at least one sample image and the at least one conditional feature information determined by the determination unit to obtain an image generation model. 23. The device according to claim 22, characterized in that the determining unit is specifically configured to: In a case where the type of the image generation task indicated by the control condition is a text image task, determining the at least one item of text feature information as the at least one item of condition feature information; In a case where the type of the image generation task indicated by the control condition is an image processing task, determining the at least one item of image feature information and the at least one item of text feature information as the at least one item of conditional feature information; The image processing task includes at least one of the following: an image-to-image task, a text-to-image task, an image elimination task, an image local modification task, and an image expansion task. 24. The device according to claim 22, characterized in that the device further comprises: a processing unit, configured to perform noise addition processing on the at least one sample image based on a Gaussian noise matrix and the mask image of the at least one sample image acquired by the acquisition unit, so as to obtain at least one noise image; The training unit is configured to train the initial model based on the at least one condition feature information determined by the determining unit and the at least one noise image processed by the processing unit to obtain the image generation model; Among them, each noise image corresponds to a conditional feature information. 25. The device according to claim 24, characterized in that the processing unit is specifically used for: Performing a weighted summation on the Gaussian noise matrix and the mask image of the at least one sample image acquired by the acquisition unit to obtain at least one noise matrix; and performing a weighted summation on the at least one noise matrix and the at least one sample image acquired by the acquisition unit to obtain at least one noise image; Among them, each noisy image corresponds to a noise matrix and a sample image. 26. The device according to claim 22, characterized in that the device further comprises: A compression unit, configured to perform feature compression on the at least one sample image acquired by the acquisition unit through an image encoder to obtain at least one image feature matrix; a processing unit, configured to perform noise addition processing on the at least one image feature matrix based on the Gaussian noise matrix and the mask image of the at least one sample image acquired by the acquisition unit, so as to obtain at least one noise image; The training unit is used to train the initial model based on the at least one condition feature information determined by the determination unit and the at least one noise image acquired by the acquisition unit to obtain an image generation model. 27. The device according to claim 26, characterized in that the compression unit is specifically used for: Performing feature compression on the at least one sample image acquired by the acquisition unit through a convolution module in the image encoder to obtain at least one compressed image feature matrix, wherein the number of pixel points in each compressed image feature matrix is greater than the number of pixel points in each image feature matrix; Through the self-attention module in the image encoder, feature extraction is performed on each compressed image feature matrix based on the correlation relationship between the pixels in each compressed image feature matrix to obtain at least one image feature matrix. 28. The device according to claim 22, characterized in that the acquisition unit is further used for: Acquire at least two sample images, image description information of each sample image, and annotation information of at least two mask areas of each sample image, wherein the at least two mask areas of each sample image are obtained by predicting the mask areas of each sample image using a segmentation model and segmenting each sample image based on the prediction results, and the annotation information of each mask area includes at least one of the following: the area of each mask area, the prediction accuracy of each mask area, and the prediction stability of each mask area; the prediction accuracy represents the prediction accuracy of the mask area using the segmentation model, and the prediction stability represents the prediction stability of the mask area using the segmentation model; Based on the annotation information of at least two mask areas of each sample image, determining a first mask area whose annotation information satisfies a filtering condition from the at least two mask areas of each sample image, wherein the filtering condition includes at least one of the following: an area of the mask area is within a preset range, a prediction accuracy of the mask area is greater than an accuracy threshold, and a prediction stability of the mask area is greater than a stability threshold; Adding the description information of the first mask area of each sample image to the image description information of each sample image to obtain a description text of each sample image; Each sample image, the description text of each sample image and the mask image of each sample image are constructed into a training sample group to obtain at least two training sample groups, wherein the mask image of each sample image is the mask image corresponding to the first mask area in each sample image. 29. The device according to claim 22, characterized in that the acquisition unit is further used to: Obtain at least two sample images and description text of each sample image; generating a second mask region in each sample image according to the size range and size information of each sample image; Each sample image, the description text of each sample image, and the mask image of each sample image are constructed into a training sample group to obtain the at least two training sample groups, wherein the mask image of each sample image is the mask image of the second mask area in each sample image. 30. The device according to claim 22, characterized in that the encoding unit is specifically used to: By using the image segmentation module in the first conditional encoder, each sample image acquired by the acquisition unit is segmented into N sample image blocks, and the pixel points of each sample image block are rearranged into a one-dimensional vector to obtain N one-dimensional vectors of each sample image, and the N one-dimensional vectors of each sample image are spliced to obtain a pixel rearranged image of each sample image, where N is an integer greater than 1; Extracting feature information of each pixel in each pixel rearrangement image through a mapping layer in the first conditional encoder to obtain at least one pixel rearrangement feature vector; Through the self-attention module in the first conditional encoder, based on the correlation relationship between pixels in each pixel rearrangement image, feature extraction is performed on each pixel rearrangement feature vector to obtain image feature information of each sample image. 31. The device according to claim 22, characterized in that the encoding unit is specifically used to: By means of a word segmenter in a second conditional encoder, each description text acquired by the acquisition unit is segmented into at least two word segments, and each word segment is encoded to obtain a text feature vector of each description text; Through the self-attention module in the second conditional encoder, based on the association relationship between the segmented words in each description text, feature extraction is performed on the text feature vector of each description text to obtain text feature information of each description text. 32. The device according to claim 24 or 26, characterized in that the training unit is specifically used for: Acquire feature information of the at least one noise image through a diffusion model, and fuse the at least one condition feature information with the feature information of the at least one noise image to obtain at least one fused feature information, where each fused feature information corresponds to one condition feature information and one piece of feature information of the noise image; Determining a noise matrix of the at least one noisy image based on the at least one fused feature information and at least one noise vector by means of the diffusion model, each noise vector representing a noise level of a noisy image; Determine a noise loss value of the at least one sample image based on a noise matrix of the at least one noise image, the Gaussian noise matrix, and the mask image of the at least one sample image acquired by the acquisition unit; Based on the noise loss value of the at least one sample image, the model parameters of the initial model are adjusted to obtain an image generation model. 33. The device according to claim 32, characterized in that the device further comprises: The extraction unit is used to obtain the feature information of the at least one noise image through the diffusion model, and fuse the at least one conditional feature information with the feature information of the at least one noise image to obtain at least one fused feature information, divide each noise image into M noise image blocks through the image segmentation module in the diffusion model, where M is an integer greater than 1; and extract the feature information of each noise image block through the linear transformation operator in the diffusion model, and add the position encoding vector to obtain the feature information of each noise image. 34. The device according to claim 32, characterized in that the training unit is specifically used for: Performing linear transformation processing on the at least one item of conditional feature information by using a linear transformation operator in the diffusion model to obtain at least one item of conditional feature information after linear transformation; The at least one linearly transformed conditional feature information and the feature information of the at least one noise image are spliced by a splicing operator in the diffusion model to obtain at least one fused feature information. 35. The device according to claim 32, characterized in that the training unit is specifically used for: By using the diffusion self-attention module in the diffusion model, feature extraction is performed on the at least one noise-added vector to obtain at least one set of scaling factors, and based on the at least one set of scaling factors, feature scaling is performed on the at least one fused feature information to obtain at least one scaled feature information, where each scaled feature information corresponds to a set of scaling factors and one fused feature information; Normalizing the at least one item of scaling feature information by a layer normalization operator in the diffusion model to obtain at least one item of noise feature information; The at least one noise feature information is subjected to channel number reduction processing by using a convolution operator in the diffusion model to obtain a noise matrix of the at least one noise image. 36. The device according to claim 32, characterized in that the training unit is specifically used for: Determining at least one first loss value based on the Gaussian noise matrix and the noise matrix of the at least one noise image, and determining at least one second loss value based on the mask image of the at least one sample image and the noise matrix of the at least one noise image; The at least one first loss value and the at least one second loss value are weightedly summed to obtain a noise loss value of the at least one sample image. 37. The device according to claim 22, characterized in that the device further comprises: an execution unit, configured to, after training an initial model based on the at least one sample image acquired by the acquisition unit, the mask image of the at least one sample image, and the at least one conditional feature information determined by the determination unit to obtain an image generation model, input the original information into the image generation model and output a derived image, or input the original information into the image generation model, perform image processing, and output a derived image; The original information includes at least one of the following: an image to be processed, a description text, and a mask image. 38. The device according to claim 37 is characterized in that the execution unit is specifically used to, when the original information includes the image to be processed, the description text and the mask image, encode the image to be processed through the first conditional encoder in the image generation model to obtain image feature information of the image to be processed, and encode the description text through the second conditional encoder in the image generation model to obtain text feature information of the description text, and perform noise processing on the image to be processed based on the Gaussian noise matrix and the mask image through the noise addition module in the image generation model to obtain a noise image; and perform image processing based on the image feature information, the text feature information and the noise image through the diffusion model in the image generation model to obtain a derivative image. 39. The device according to claim 38, characterized in that the device further comprises: A compression unit, configured to perform feature compression on the image to be processed by using an image encoder in the image generation model to obtain an image feature matrix before performing noise processing on the image to be processed by using a noise adding module in the image generation model based on a Gaussian noise matrix and the mask image to obtain a noise image; A processing unit, configured to perform noise addition processing on the image feature matrix based on a Gaussian noise matrix and the mask image through a noise addition module in the image generation model to obtain the noise image; A decoding unit is used to decode the derivative image obtained by the execution unit through an image decoder in the image generation model, and output the decoded derivative image. 40. An electronic device, characterized in that it comprises a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the training method of the image generation model as described in any one of claims 4-21 are implemented.