CN115070753B - A multi-objective reinforcement learning method for unsupervised image editing - Google Patents

Abstract

The present invention relates to a multi-objective reinforcement learning method based on unsupervised image editing, comprising: obtaining a multi-objective task data set about a robot control scene; training a generative adversarial network and a feature space encoder to decouple factors in the image that are highly relevant and irrelevant to the task; performing singular value decomposition on the weights of the fully connected layer corresponding to each subspace, obtaining several feature vectors with the largest contribution as editable directions with semantic information, and training the editable direction encoder to identify the category and scale of the editable direction; obtaining an editable representation space of the image based on the output of the editable direction encoder as the input of the control strategy network and the calculation of the reward function, and training the robot by controllably sampling various target tasks in the editable representation space, and finally obtaining a control strategy that can complete multiple goals. Compared with the prior art, the present invention has the advantages of being able to unsupervisedly decouple task-related factors, improve sample efficiency and generalization performance, and the like.

Description

A multi-objective reinforcement learning method for unsupervised image editing

Technical Field

The present invention relates to the technical field of intelligent agent autonomous action learning, and in particular to a multi-objective reinforcement learning method based on unsupervised image editing.

Background Art

With the rise of artificial intelligence algorithms and the rapid development of various hardware devices, robotics has played an important role in many fields such as medical care, service, assembly, security, rescue, and transportation. However, customized deployment of robots for a single task is time-consuming and laborious. Therefore, how to let robots learn a control strategy that can achieve multiple goals has always been our goal.

Traditional robot control methods are extremely dependent on the professional knowledge and software programming level of technicians, and different robot trajectory distributions need to be designed for specific task objectives. When the application scenario changes, the previously deployed robot skills cannot be reused. Deep reinforcement learning methods can enable robots to autonomously learn task-related skills through exploration and utilization. However, this learning method requires manual design of corresponding reward functions for different target tasks, and the learned control strategies can only complete specific target tasks and cannot be generalized to tasks with similar environment and task structures but different target positions. Multi-objective reinforcement learning methods give robots the ability to complete various different target tasks by simultaneously inputting current observations and task objectives into the control strategy. However, this type of algorithm uses images as input, the training process is complex and difficult to converge, and the data utilization rate is low. It may take hundreds or thousands of hours to collect data and train strategies in a simulation environment, which is impractical in real scenarios.

In daily life and work, humans are often able to decouple the various components of the environment and complete various corresponding target actions based on the state changes of key parts. How to decouple and represent the key objects in the environment image in an unsupervised way remains a challenge.

Summary of the invention

The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a multi-objective reinforcement learning method based on unsupervised image editing that can improve sample efficiency and generalization performance.

The purpose of the present invention can be achieved by the following technical solutions:

A multi-objective reinforcement learning method based on unsupervised image editing, the multi-objective reinforcement learning method comprising:

Step 1: Obtain a multi-objective task dataset for robot control scenarios;

Step 2: Train the generative adversarial network and feature space encoder to divide the latent variable space into three subspaces, corresponding to task-irrelevant factors, the robot, and the manipulated object, to decouple the various parts of the image;

Step 3: Perform singular value decomposition on the weights of the fully connected layer corresponding to each subspace to obtain several eigenvectors with the largest contribution as editable directions with semantic information. The latent variables can be increased or decreased along the editable directions to perform image editing, and an editable direction encoder is trained to identify the category and scale of the editable directions.

Step 4: Based on the output of the editable direction encoder, an editable representation space of the image is obtained as the input of the control strategy network and the calculation of the reward function. At the same time, the robot is trained by controllably sampling various target tasks in the editable representation space, and finally a control strategy that can achieve multiple goals is obtained.

Preferably, the step 1 is specifically:

Build a robot virtual simulation environment, and control the robot to complete multiple target tasks through sampling control strategies, and create a multi-target task dataset for robot control scenarios.

More preferably, the multi-objective task includes two tasks: grabbing wooden blocks and assembling. The robot virtual simulation environment includes a UR5 robotic arm, an operating table and a number of operating objects. The robot is controlled to interact with the environment by sampling action instructions from a random exploration strategy to obtain various image sequence data for each task, and the target position of each sequence is obtained by sampling from a uniform distribution.

Preferably, the generative adversarial network comprises:

The generative network is expected to generate sufficiently realistic images that the discriminative network cannot distinguish between the real and the fake;

The discriminative network is expected to distinguish whether the image is real or generated by the generative network;

The latent variables randomly sampled from the Gaussian distribution are mapped into three W feature spaces through 8 fully connected layers, where _W1 corresponds to task-independent factors and the other two subspaces correspond to task-related factors, where _W2 corresponds to the robot and _W3 corresponds to the object to be operated.

The feature space encoder is used to encode the image generated by the generating network back to the features in the W space;

In step 2, the image generated by the generative network is used as the input of the feature space encoder, and the generative network input corresponding to each image is used as the output label of the feature space encoder, so as to supervise the training of the feature space encoder.

Preferably, the method for decoupling the various parts of the image in step 2 is:

The various parts of the image are decoupled by exchanging the latent variables of one subspace and keeping the latent variables of other subspaces unchanged.

More preferably, the method of decoupling the various parts of the image in step 2 is specifically as follows:

The two images in the multi-target task dataset are taken as a set of data, and the corresponding W spatial features are recorded as W _1,2,3 and W _a,b,c ;

Exchange the features of a certain W subspace with each other, and keep the other features unchanged, and obtain two new features W _1,b,3 and W _a,2,c ;

Input the new features into the generative network to obtain two new images;

The two generated images are encoded by the feature space encoder, and the feature exchange operation is performed again to obtain new features W′ _1,2,3 and W′ _a,b,c , and the root mean square error is used to supervise the feature, specifically:

Preferably, the fully connected layer parameters corresponding to each subspace in step 3 are set to is a non-square matrix;

Then the affine transformation F of the fully connected layer for the hidden variable z obtained by random sampling is:

W＝F(z)＝Az+b

By adding a direction N with semantic information to the corresponding W spatial feature, the generated image can be controllably edited so that the generated image changes in a certain content factor.

Preferably, the singular value decomposition in step 3 is specifically as follows:

Singular value decomposition is used to perform matrix decomposition on the fully connected layer parameters _Ai :

A _i = U∑V ^T

Among them, U is an m*m square matrix; ∑ is an m*n matrix, all elements except the elements on the main diagonal are 0, and each element on the main diagonal is called a singular value; V is an n*n square matrix;

Then, the singular values in ∑ are sorted according to their sizes to obtain the first several singular values with the largest contribution values. The corresponding feature vector is the direction N with semantic information, that is, the editable direction N of the image is learned in an unsupervised way.

Preferably, the editable direction encoder is specifically:

The editable direction encoder is used to encode the changes of the two images in the W space of the adversarial generation network, including the category _Ni and scale α of the editable direction. Step 3 performs controllable editing on the generated image based on the editable direction N to obtain image pairs and corresponding editing categories and scales, thereby performing supervised training on the editable direction encoder.

Preferably, the step 4 is specifically:

The control policy network takes the output of the editable direction encoder as the embedding space input of the observation image and task goal, and outputs the joint angles of the robot to control the robot to complete the task.

Compared with the prior art, the present invention has the following beneficial effects:

Improved sample efficiency and generalization performance of multi-objective reinforcement learning algorithm: The multi-objective reinforcement learning method based on unsupervised image editing in the present invention uses the editable direction category and scale of the image as the encoding input control strategy of the image, which more clearly and efficiently reflects the state changes of each element in the image. The control strategy network is trained through the reinforcement learning algorithm PPO to output appropriate robot actions, so that the robot can complete tasks with various different goals, and greatly improves the sample efficiency and generalization performance of the multi-objective reinforcement learning algorithm.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of a flow chart of a multi-objective reinforcement learning method for unsupervised image editing according to an embodiment of the present invention;

FIG2 is a schematic diagram of a robot virtual simulation environment according to an embodiment of the present invention;

FIG3 is a schematic diagram of the structure of a generative adversarial network according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of the network structure of an editable direction encoder according to an embodiment of the present invention.

DETAILED DESCRIPTION

The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

This embodiment takes the Universal Robots UR5 robotic arm as an example. Its control workstation is installed with Ubuntu 16.04 system, equipped with Intel Core i7-10700K, 8 cores and 16 threads, turbo frequency 5.1GHz, GPU is NVIDIA GTX1080*2, memory is 32GDDR4 memory, and there is also a general RGB camera to observe the overall scene of the operating console.

A multi-objective reinforcement learning method based on unsupervised image editing, the process of which is shown in Figure 1, includes:

Step 1: Build a robot simulation environment based on OpenAI Gym and Mujoco simulation platforms. The simulation environment diagram is shown in Figure 2. In the built virtual environment, collect a multi-objective task dataset for robot control scenarios, including two tasks: grabbing wooden blocks and assembling. The environment includes a UR5 robotic arm, an operating table, and several operating objects. By sampling action instructions from the random exploration strategy to control the robot to interact with the environment, various image sequence data for each task are obtained, and the target position of each sequence is sampled from a uniform distribution.

Step 2: Based on the existing dataset, train a generative adversarial network and a feature space encoder. The generative adversarial network consists of a generative network and a discriminative network. The discriminative network is expected to be able to distinguish whether the image is real or generated by the generative network. The generative network is expected to generate sufficiently real images that the discriminative network cannot distinguish between true and false. The feature space encoder aims to encode the images generated by the generative network back to features in the W space, and can be supervised by randomly generated samples from the generative network.

The structure of the adversarial generative network is shown in Figure 3, where the generative network consists of 3 convolutional layers with a convolution size of 5, a stride of 0.5, and an activation function of ReLU, and the discriminative network consists of 3 convolutional layers with a convolution size of 5, a stride of 0.5, and an activation function of ReLU, 2 fully connected layers, and an activation function layer. The latent variables randomly sampled from the Gaussian distribution are mapped to 3 W feature spaces through 8 fully connected layers, where _W1 corresponds to task-independent factors, including the lighting of the simulation environment, the color of the desktop, and the color of the wall. The other two subspaces correspond to task-related factors, where _W2 corresponds to the robot and _W3 corresponds to the object to be operated.

The two images in the multi-target task dataset are taken as a set of data, and the corresponding W spatial features are recorded as W _1,2,3 and W _a,b,c ;

Exchange the features of a certain W subspace with each other, and keep the other features unchanged, and obtain two new features W _1,b,3 and W _a,2,c ;

Input the new features into the generative network to obtain two new images;

The two generated images are encoded by the feature space encoder, and the feature exchange operation is performed again to obtain new features W′ _1,2,3 and W′ _a,b,c , and the root mean square error is used to supervise the feature, specifically:

This ensures that the three W subspaces can be decoupled, and changes in the characteristics of one subspace will not affect the image content corresponding to other subspaces.

Step 3: Weights of the fully connected layer corresponding to each subspace Perform singular value decomposition and obtain the eigenvectors with the largest contribution as the editable directions N with semantic information. Set the parameters of the fully connected layer corresponding to each subspace to is a non-square matrix. Then the affine transformation F of the fully connected layer for the hidden variable z obtained by random sampling can be expressed as follows:

W＝F(z)＝Az+b

By adding the direction N with semantic information to the corresponding W spatial feature, the generated image can be edited in a controllable manner so that the generated image changes in a certain content factor, such as changing the position, shape, ambient lighting and color of the operating object.

Non-square The singular value decomposition can be expressed as:

A _i = U∑V ^T

Among them, U is an m*m square matrix; ∑ is an m*n matrix, all elements except the elements on the main diagonal are 0, and each element on the main diagonal is called a singular value; V is an n*n square matrix;

Then, the singular values in ∑ are sorted according to their sizes to obtain the top 10 singular values with the largest contribution values. The corresponding eigenvector is the direction N with semantic information, that is, the editable direction N of the image is learned in an unsupervised way.

In addition, according to the learned editable direction N, an editable direction encoder Enc is learned, as shown in Figure 4. First, the latent variable z _i is randomly sampled, and the feature W in the corresponding W space is obtained through two layers of fully connected layers FC, and the corresponding image I _i is generated through the generative network Gen. Then an editable direction N _i is uniformly sampled from the K editable directions, and the editing scale α~U{-ε,ε} is randomly sampled. The two are multiplied and added to w, and the image I _j is generated through the generative network. The images I _i and I _j are input into the editable direction encoder Enc together, and the corresponding editing category N _i and scale α are predicted. A large number of samples and real labels are generated by the generator Gen, so that Enc can be supervised for training.

S4: The editable direction encoder is used to obtain the edit category N _t and scale α _t between the robot's initial observation image obs _{t = 0} and the current frame image obs _t , as well as the edit category N _goal and scale α _goal between obs _{t = 0} and the task goal image obs _goal . (N _t , α _t , N _goal , α _goal ) is used as the input of the control strategy network, and the joint angle of the robot is finally output to control the robot to complete the task. The specific expression of the control strategy network is as follows:

π=(A|·)=π(A|N _t ,α _t ,N _goal ,α _goal )

Based on the output of the editable direction encoder, the editable representation space of the image is obtained as the input of the control strategy and the calculation of the reward function. At the same time, the robot is trained by controllably sampling various target tasks in the editable representation space, and finally a control strategy that can complete multiple goals is obtained. The present invention uses the editable direction category and scale of the image as the encoding input control strategy of the image, which more clearly and efficiently reflects the state changes of each element in the image. The control strategy network is trained to output appropriate robot actions through the reinforcement learning algorithm PPO, so that the robot can complete tasks with various different goals, and greatly improves the sample efficiency and generalization performance of the multi-objective reinforcement learning algorithm.

The innovation of the present invention is:

(I) A cyclic exchange operation method based on hidden variables in a generative adversarial network is proposed to decouple task-related factors from task-irrelevant factors in the robot operation scenario, so that the control strategy can focus on the state changes of task-related factors and reduce the redundancy of the robot's observation representation.

(II) An unsupervised image editing algorithm is proposed to perform singular value decomposition on the parameters of different factors in the control scene in latent space, and obtain a series of semantically editable directions, thereby achieving controllable generation of each object in the image. Based on the learned series of editable directions, an editable direction encoder is trained to extract the structured representation of each object.

(III) A multi-objective control strategy based on image editing is proposed. The output of the encoder is used to represent the robot observation and target task images, which can greatly shorten the training time of the algorithm. The structured representation with semantic information is used to improve the sample efficiency and generalization performance of the multi-objective reinforcement learning algorithm.

The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims (6)

1. A multi-objective reinforcement learning method based on unsupervised image editing, characterized in that the multi-objective reinforcement learning method comprises: Step 1: Obtain a multi-objective task dataset for robot control scenarios; Step 2: Train the generative adversarial network and feature space encoder to divide the latent variable space into three subspaces, corresponding to task-irrelevant factors, the robot, and the manipulated object, to decouple the various parts of the image; Step 3: Perform singular value decomposition on the weights of the fully connected layer corresponding to each subspace to obtain several feature vectors with the largest contribution as editable directions with semantic information, and train an editable direction encoder to identify the category and scale of the editable direction; Step 4: Based on the output of the editable direction encoder, an editable representation space of the image is obtained as the input of the control strategy network and the calculation of the reward function. At the same time, the robot is trained by controllably sampling various target tasks in the editable representation space, and finally a control strategy that can achieve multiple goals is obtained; The method for decoupling the various parts of the image in step 2 is: Decouple the various parts of the image by exchanging one subspace latent variable and keeping the other subspace latent variables unchanged; The method for decoupling the various parts of the image in step 2 is specifically as follows: The two images in the multi-target task dataset are taken as a set of data, and the corresponding W spatial features are recorded as W _1,2,3 and W _a,b,c ; Exchange the features of a certain W subspace with each other, and keep the other features unchanged, and obtain two new features W _1,b,3 and W _a,2,c ; Input the new features into the generative network to obtain two new images; The two generated images are encoded by the feature space encoder, and the feature exchange operation is performed again to obtain new features W ₁ ' _{, 2, 3} and W _a ' _{, b, c} , and the root mean square error is used to supervise the feature, specifically: The singular value decomposition in step 3 is specifically as follows: Singular value decomposition is used to perform matrix decomposition on the fully connected layer parameters _Ai : A _i ＝UΣV ^T Among them, U is an m*m square matrix; Σ is an m*n matrix, all elements except the elements on the main diagonal are 0, and each element on the main diagonal is called a singular value; V is an n*n square matrix; Then, the singular values in Σ are sorted according to their sizes to obtain the first several singular values with the largest contribution values. The corresponding feature vectors are the directions N with semantic information, that is, the editable directions N of the image are learned in an unsupervised way. The editable direction encoder is specifically: The editable direction encoder is used to encode the changes of two images in the W space of the adversarial generative network, including the category _Ni and scale α of the editable direction. Based on the linear weighted operation of the editable direction in the W space, the generated image is controllably edited to obtain image pairs and the corresponding editing categories and scales, thereby performing supervised training on the editable direction encoder. 2. According to the multi-objective reinforcement learning method based on unsupervised image editing according to claim 1, it is characterized in that the step 1 is specifically: Build a robot virtual simulation environment, and control the robot to complete multiple target tasks through sampling control strategies, and create a multi-target task dataset for robot control scenarios. 3. According to claim 2, a multi-objective reinforcement learning method based on unsupervised image editing is characterized in that the multi-objective tasks include two tasks: grabbing wooden blocks and assembling; the robot virtual simulation environment includes a UR5 robotic arm, an operating table and a number of operating objects; the robot is controlled to interact with the environment by sampling action instructions from a random exploration strategy to obtain various image sequence data for each task, and the target position of each sequence is obtained by sampling from a uniform distribution. 4. According to the multi-objective reinforcement learning method based on unsupervised image editing in claim 1, it is characterized in that the adversarial generative network comprises: The generative network is expected to generate sufficiently realistic images that the discriminative network cannot distinguish between the real and the fake; The discriminative network is expected to distinguish whether the image is real or generated by the generative network; The latent variables randomly sampled from the Gaussian distribution are mapped to three W feature spaces through 8 fully connected layers, where W1 corresponds to task-independent factors and the other two subspaces correspond to task-related factors, where W2 corresponds to the robot and W3 corresponds to the object to be operated. The feature space encoder is used to encode the image generated by the generating network back to the features in the W space; The image generated by the generative network is used as the input of the feature space encoder, and the generative network input corresponding to each image is used as the output label of the feature space encoder, so as to supervise the training of the feature space encoder; 5. The multi-objective reinforcement learning method based on unsupervised image editing according to claim 1, characterized in that the fully connected layer parameters corresponding to each subspace in step 3 are set to is a non-square matrix; Then the affine transformation F of the fully connected layer for the hidden variable z obtained by random sampling is: W＝F(z)＝Az+b By adding a direction N with semantic information to the corresponding W spatial feature, the generated image can be controllably edited so that the generated image changes in a certain content factor. 6. The multi-objective reinforcement learning method based on unsupervised image editing according to claim 1, characterized in that the step 4 is specifically: The control policy network takes the output of the editable direction encoder as the embedding space input of the observation image and task goal, and outputs the joint angles of the robot to control the robot to complete the task.