CN117975190B - Method and device for processing simulated learning mixed sample based on vision pre-training model - Google Patents

Abstract

The present invention provides a mixed sample processing method and device for imitation learning based on a visual pre-training model, the method comprising: obtaining an expert sample set; adding target noise to a suboptimal expert sample to obtain a noise expert sample, and obtaining a mixed sample set according to the noise expert sample and the optimal expert sample; calibrating the weight coefficient of the mixed sample set, predicting and scoring the redistributed mixed sample set, and then training a strategy network and a reward function network according to the scoring results, scoring each sample of an evaluation data set according to the target reward function network, and obtaining a prediction ranking corresponding to the evaluation data set, so as to update the weight coefficient corresponding to each sample in the redistributed mixed sample set, and finally imitating learning the redistributed weight coefficient according to the target strategy network to obtain an optimized expert sample. The method of the present invention performs differentiated learning on mixed expert samples of different qualities, improves the sample distribution of the data set, and enhances the generalization ability of the imitation learning agent.

Description

Imitation learning mixed sample processing method and device based on visual pre-training model

Technical Field

The present invention relates to the field of deep learning technology, and in particular to a method and device for processing mixed samples of imitation learning based on a visual pre-training model.

Background Art

With the continuous development of high-tech technologies such as robots, self-driving cars, and game agents, how to enable them to complete complex decision-making tasks and quickly adapt to environmental changes has become an important research issue.

Adversarial generative imitation learning is one of the representative algorithms of imitation learning. It draws on the idea of Generative Adversarial Network (GAN) and makes the agent close to the decision-making ability of expert agents through adversarial training of agents and reward functions. Adversarial generative imitation learning aims to imitate the behavior of experts by training a generative model to achieve automatic task solving. In this method, the generative model is trained to generate samples similar to expert behavior, while the discriminative model is used to distinguish between samples generated by the generative model and real samples of experts, so as to enable the generative model to gradually approach expert behavior.

In the related technology, adversarial generative imitation learning relies on high-quality data. In many task scenarios, due to the limitation of labor costs, there is a lack of sufficient optimal expert samples for the imitation learning agent to be fully trained. In addition, common expert data sets usually contain suboptimal expert samples. Imitation of suboptimal expert samples will reduce the performance of the imitation learning algorithm, resulting in low efficiency in processing related characters. In addition, the data in the expert data set has poor generalization ability in high-dimensional state space. With the increase of the dimension of the state space, for example, in the visual observation task scenario (the state is represented by a high-dimensional image), the redundant information about the environment increases significantly, making it difficult to capture the changes in the main body of the agent. In this case, the converged agent will be significantly affected by the environmental changes when it is placed in a new test environment, resulting in a decrease in the generalization ability and learning ability of the agent.

Summary of the invention

The present invention provides a sample processing method and device based on an adversarial generative imitation learning algorithm, which is used to solve the problems that the adversarial generative imitation learning algorithm in the prior art relies on high-quality data, but lacks high-quality data, and is affected by low-quality data, resulting in the inability to fully train the imitation learning agent, resulting in poor training effect. In addition, the generalization ability of the existing expert data set in high-dimensional state space is poor, resulting in low generalization ability of the trained agent. The method improves the quality of the expert sample set, thereby improving the generalization ability of the imitation learning agent.

The present invention provides a mixed sample processing method for imitation learning based on a visual pre-training model, comprising:

Acquire an expert sample set, wherein the expert sample set includes an optimal expert sample and a suboptimal expert sample;

Adding target noise to the suboptimal expert samples to obtain noise expert samples, and processing the noise expert samples and the optimal expert samples according to a generative adversarial network to obtain a mixed sample set;

Calibrate the weight coefficient of each sample in the mixed sample set to obtain a redistributed mixed sample set, predict the redistributed mixed sample set according to the policy network to obtain an action prediction result; score the action prediction result according to the reward function network to obtain a scoring result; train the policy network and the reward function network according to the discriminant loss function and the scoring result to obtain a target policy network and a target reward function network;

According to the target reward function network, each sample in the evaluation data set is scored to obtain the predicted ranking corresponding to the evaluation data set, the ranking error loss is calculated according to the predicted ranking, the weight coefficient corresponding to each sample in the redistributed mixed sample set is updated by gradient optimization to obtain the redistributed weight coefficient, and the redistributed weight coefficient is imitated and learned according to the target strategy network to obtain the optimized expert sample; the evaluation data set belongs to the mixed sample set.

The present invention also provides a mixed sample processing method for imitation learning based on a visual pre-training model, wherein obtaining an expert sample set comprises:

Based on the visual pre-training model, forward graph reasoning is performed on each sample in the expert sample set, and the feature graph corresponding to the middle layer of the network is determined to be the effective feature of the expert sample set.

The present invention also provides an imitation learning mixed sample processing method based on a visual pre-training model, wherein the adversarial generative network includes a generative network and a discriminative network;

The processing of the noise expert samples and the optimal expert samples according to the generative adversarial network to obtain a mixed sample set includes:

Extracting features from the noise expert sample and the optimal expert sample according to the generating network to obtain a state representation vector;

Performing feature discrimination on the source of the state representation vector according to the discrimination network to obtain a discrimination result;

Calculating the loss value of the generating network according to the discrimination result and the first loss function, and optimizing the parameters of the generating network according to the loss value of the generating network to obtain the optimized generating network; calculating the loss value of the discriminant network according to the discrimination result and the second loss function, and optimizing the parameters of the discriminant network according to the loss value of the discriminant network to obtain the optimized discriminant network to output the mixed sample set;

Among them, the first loss function is determined based on the occupancy measurement of the optimal expert sample state-action pair, the occupancy measurement after normal distribution noise, the state characterization vector, the conditional probability distribution corresponding to the state characterization vector, and the discriminant network parameters; the second loss function is determined based on the occupancy measurement of the optimal expert sample state-action pair, the occupancy measurement after normal distribution noise, the state characterization vector, the conditional probability corresponding to the state characterization vector and the generation network parameters.

The present invention also provides a mixed sample processing method for imitation learning based on a visual pre-training model, wherein the first loss function includes:

The second loss function includes:

Wherein, E is the expectation, D(z) is the discriminant network, ρ ^* (s, a) is the occupancy measure of the optimal expert sample state-action pair, ρ′(s, a) is the occupancy measure after adding normal distribution noise, z is the state representation vector, p(z|x) is the conditional probability distribution corresponding to the state representation vector, θ ^D is the discriminant network parameter, and θ ^E is the generative network parameter.

The present invention also provides a mixed sample processing method for imitation learning based on a visual pre-training model, wherein the first loss function further includes:

A target regularization term, wherein the target regularization term is determined according to the KL divergence corresponding to the optimal expert sample and the state representation vector;

The second loss function also includes: the target regularization term.

The present invention also provides a mixed sample processing method for imitation learning based on a visual pre-training model, wherein the evaluation data set is obtained by sorting the expert sample set according to a priori;

The calculating of the sorting error loss according to the predicted sorting and updating the weight coefficient corresponding to each sample in the redistributed mixed sample set by gradient optimization to obtain the weight coefficient after redistribution include:

Calculate the ranking difference loss through the predicted ranking, the prior ranking and the ranking loss function;

Determining the weight coefficient after redistribution according to the sorting difference loss;

The ranking loss function includes:

Among them, i and j are sample numbers, η _ζi is the true cumulative expected return of sample No. i, η′ _ζi is the predicted cumulative expected return of sample No. i; η _ζj is the true cumulative expected return of sample No. j, η′ _ζj is the predicted cumulative expected return of sample No. j.

The present invention also provides a mixed sample processing device for imitation learning based on a visual pre-training model, comprising:

A sample acquisition module, used to acquire an expert sample set, wherein the expert sample set includes an optimal expert sample and a suboptimal expert sample, and each sample in the expert sample set corresponds to a different weight coefficient;

A first processing module is used to add target noise to the suboptimal expert samples to obtain noise expert samples, and process the noise expert samples and the optimal expert samples according to a generative adversarial network to obtain a mixed sample set;

The second processing module is used to calibrate the weight coefficient of each sample in the mixed sample set to obtain a redistributed mixed sample set, predict the redistributed mixed sample set according to the policy network to obtain an action prediction result; score the action prediction result according to the reward function network to obtain a scoring result; train the policy network and the reward function network according to the discriminant loss function and the scoring result to obtain a target policy network and a target reward function network;

The third processing module is used to score each sample in the evaluation data set according to the target reward function network, obtain the predicted ranking corresponding to the evaluation data set, calculate the ranking error loss according to the predicted ranking, update the weight coefficient corresponding to each sample in the redistributed mixed sample set through gradient optimization, obtain the redistributed weight coefficient, and perform imitation learning on the redistributed weight coefficient according to the target strategy network to obtain the optimized expert sample; the evaluation data set belongs to the mixed sample set.

The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the mixed sample processing method for imitation learning based on a visual pre-training model as described in any one of the above is implemented.

The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described methods for processing mixed samples by imitation learning based on a visual pre-training model.

The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the above-mentioned methods for processing mixed samples by imitation learning based on a visual pre-training model.

The present invention provides an imitation learning mixed sample processing method and device based on a visual pre-training model, which improves the data distribution of suboptimal expert samples by adding target noise to suboptimal expert samples, optimizes the parameters of the adversarial generative network according to the noise expert samples and the optimal expert samples, obtains a mixed sample set, improves the feature distribution of the suboptimal expert samples, and then calibrates the weight coefficient of each sample in the mixed sample set, predicts and scores the calibrated mixed sample set according to the policy network to update the policy network and the reward function network, scores each sample in the evaluation data set according to the target reward function network to obtain the corresponding prediction ranking, and then updates the weight coefficient corresponding to each sample in the redistributed mixed sample set according to the prediction ranking to obtain the redistributed weight coefficient, and imitates the redistributed weight coefficient according to the target policy network to obtain optimized expert samples, which can perform differentiated learning for mixed expert samples of different qualities, improve the sample distribution of the data set, and enhance the generalization ability of the imitation learning agent.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

FIG1 is a flow chart of a mixed sample processing method for imitation learning based on a visual pre-training model provided by the present invention;

FIG2 is a second flow chart of the mixed sample processing method for imitation learning based on the visual pre-training model provided by the present invention;

3 is a schematic diagram of a process for extracting an intermediate layer feature map based on a visual pre-training model provided by the present invention;

FIG4 is a third flow chart of the mixed sample processing method for imitation learning based on the visual pre-training model provided by the present invention;

5 is a schematic diagram of the structure of a mixed sample processing device for imitation learning based on a visual pre-training model provided by the present invention;

FIG. 6 is a schematic diagram of the structure of an electronic device provided by the present invention.

DETAILED DESCRIPTION

In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

The following describes the mixed sample processing method and device for imitation learning based on a visual pre-training model of the present invention in conjunction with Figures 1 to 5.

FIG1 is a flow chart of a mixed sample processing method for imitation learning based on a visual pre-training model provided by the present invention. As shown in FIG1 , the mixed sample processing method for imitation learning based on a visual pre-training model comprises the following steps:

Step 110: Obtain an expert sample set, where the expert sample set includes optimal expert samples and suboptimal expert samples.

In this step, the expert sample set may include at least one required training sample or test sample in the following scenarios: image processing tasks such as image classification, target detection and image segmentation.

For example, for image classification tasks, the expert sample set includes image data to be classified and its sample image data; for image segmentation tasks, the expert sample set includes image data to be segmented and its sample image data.

In this embodiment, the optimal expert sample is of higher quality than the suboptimal expert sample. For example, the sample contains richer data content and more effective information.

Step 120: Add target noise to the suboptimal expert samples to obtain noisy expert samples, and process the noisy expert samples and the optimal expert samples according to the generative adversarial network to obtain a mixed sample set.

In this step, the target noise includes normally distributed noise, such as Gaussian noise or Gaussian white noise.

In this embodiment, the generative adversarial network includes a generative network and a discriminative network; wherein the generative network uses noise expert samples and optimal expert samples as features to extract new state representations, and discriminates the source of the new state representations through the discriminative network, and calculates the corresponding loss function according to the discrimination results, thereby optimizing the parameters of the generative adversarial network and the discriminative network, and through the supervision and guidance of the feature extractor learning, the hidden layer state representation of the suboptimal expert sample approaches the representation of the optimal expert sample.

In this embodiment, the behavioral characteristics of the suboptimal expert sample are compared with the behavioral characteristics of the optimal expert sample, and a noise signal is used to simulate the difference in distribution between the two, while being compatible with the deep generative imitation learning algorithm architecture; specifically, by minimizing the distribution distance between the two distributions, such as the KL (Kullback-Leibler) divergence, the state representation of the suboptimal expert sample after feature extraction is made close to the state representation of the optimal expert sample.

Step 130: calibrate the weight coefficient of each sample in the mixed sample set to obtain a redistributed mixed sample set, predict the redistributed mixed sample set according to the policy network to obtain an action prediction result; score the action prediction result according to the reward function network to obtain a scoring result; train the policy network and the reward function network according to the discriminant loss function and the scoring result to obtain a target policy network and a target reward function network.

In this step, each expert sample corresponds to a different weight coefficient. The higher the weight coefficient corresponding to the best expert sample in the expert sample set, the higher the quality of the expert sample set, and the more suitable it is as input data for the above-mentioned image processing task.

For example, the weight coefficient β∈(0,1), if the weight coefficient of a sample is closer to 1, the quality of the sample is higher.

In this step, the imitation learning strategy is used. The closer the weight coefficient corresponding to the sample is to 1 (the larger the sample weight is), the more important it is to imitate the sample.

Figure 2 is the second flow chart of the mixed sample processing method for imitation learning based on the visual pre-training model provided by the present invention. In the embodiment shown in Figure 2, in the weight learning stage, a weight coefficient is calibrated for each sample in the mixed sample set to indicate the quality of the sample (initialized to 1, indicating that the quality of each sample is considered to be the same a priori); after calibrating the weight coefficient, the expert sample redistributed by the weight coefficient is obtained, that is, the redistributed mixed sample set.

In this embodiment, the redistributed mixed sample set is taken as input into the policy network to predict the corresponding action output.

For example, the scenario for imitation learning is a sequential decision-making task. Taking autonomous driving as an example, the expert sample is the road image obtained by the camera, and the predicted action is the steering wheel angle and throttle force.

In the embodiment shown in FIG2 , in the imitation learning stage, the action output of the policy network, that is, the action prediction result, is input into the reward function network to score different actions (for example, high scores are given to expert samples and low scores are given to samples generated by the policy network). The policy network and the reward function network are then updated by calculating the loss function to obtain the target policy network and the target reward function network, so as to improve the ability of the policy network to generate expert-like actions and to improve the ability of the reward function to distinguish samples generated by the policy network, that is, adversarial training.

Step 140: score each sample in the evaluation data set according to the target reward function network to obtain the predicted ranking corresponding to the evaluation data set, calculate the ranking error loss according to the predicted ranking, update the weight coefficient corresponding to each sample in the redistributed mixed sample set through gradient optimization to obtain the redistributed weight coefficient, perform imitation learning on the redistributed weight coefficient according to the target strategy network to obtain the optimized expert sample; the evaluation data set belongs to the mixed sample set.

In this step, the corresponding evaluation data set is extracted from the mixed sample set by sorting according to preset weight coefficients (ie, prior sorting).

For example, after obtaining the target reward function network, a small number of samples are extracted from the original mixed expert samples, and the relative quality of these samples is calibrated as prior knowledge to form an evaluation data set.

In this embodiment, the evaluation data set is obtained by prior sorting the expert sample set; the sorting error loss is calculated according to the predicted sorting update, and the weight coefficient corresponding to each sample in the mixed sample set is redistributed by gradient optimization to obtain the weight coefficient after redistribution, including: calculating the sorting difference loss through the predicted sorting, the prior sorting and the sorting loss function; determining the weight coefficient after redistribution according to the sorting difference loss; the sorting loss function includes:

Among them, i and j are sample numbers, η _ζi is the true cumulative expected return of sample No. i, η′ _ζi is the predicted cumulative expected return of sample No. i; η _ζj is the true cumulative expected return of sample No. j, η′ _ζj is the predicted cumulative expected return of sample No. j.

In the embodiment shown in FIG2 , a reward function network is used to score the data in the evaluation data set to obtain a predicted ranking. The predicted ranking and the calibrated prior ranking information can be compared to calculate the ranking difference loss, and the weight coefficient is updated with this loss function. The smaller the loss, the better the reward function is learned in the imitation learning stage, which indirectly indicates that the weight coefficient is set reasonably.

In this embodiment, after obtaining the redistributed weight coefficients, the mixed samples corresponding to the redistributed weight coefficients are sent to the strategy network for imitation learning. Imitation learning and weight learning are performed alternately until a predetermined number of iterations are reached and then the optimized expert samples are output.

In this embodiment, in a specific training process, the weight coefficients are adaptively optimized for differentiated learning of each sample in the expert sample set, and the current imitation learning strategy is redistributed by redistributing the weight coefficients of each sample. In each differentiated learning process, the optimized generation network outputs a reward value, and the weight coefficients corresponding to each sample are reordered according to the reward value and the sorting loss function by the optimized discriminant network; in multiple training processes, the adaptive optimization of the sorting of the weight coefficients corresponding to each sample is achieved by maximizing the cumulative expected reward value to improve the distribution of each sample in the expert sample set and improve the quality of the expert sample set; wherein, the maximized cumulative expected reward is the cumulative value of the expected reward predicted by the reward function after training with the adversarial inverse reinforcement learning algorithm.

Specifically, assuming that the current hybrid expert strategy is represented by π ^d , the occupancy measure of the current strategy in the process of interacting with the environment is By redistributing the state-action pairs of π ^d, we can get the new strategy π _new ; its occupancy measurement can be expressed as The imitation learning loss function after weight coefficient redistribution can be expressed as:

Among them, _Limitation can be the loss function of any traditional imitation learning algorithm such as behavior cloning or inverse reinforcement learning.

In this embodiment, by performing differentiated learning on different expert samples through weight coefficients, the performance of the imitation learning algorithm can be maximized, that is, the purpose of maximizing the cumulative expected return can be achieved; the specific optimization problem can be expressed as:

Among them, β ^* represents the optimal weight coefficient distribution, represents the cumulative expected return of the redistributed strategy π _new , expressed as:

Among them, R is the reward function of the environment, and γ is the attenuation coefficient. The purpose of weight learning is to maximize the performance of the imitation learning algorithm, so the performance of the imitation learning algorithm can be evaluated to reflect the effect of the current weight coefficient learning. Let the current time be t, s _t represents the state information of the expert sample at time t, and a _t represents the action information of the expert sample at time t.

The imitation learning mixed sample processing method based on the visual pre-training model provided by the embodiment of the present invention improves the data distribution of the suboptimal expert samples by adding target noise to the suboptimal expert samples, optimizes the parameters of the adversarial generative network according to the noise expert samples and the optimal expert samples, obtains a mixed sample set, improves the feature distribution of the suboptimal expert samples, and then calibrates the weight coefficient of each sample in the mixed sample set, predicts and scores the calibrated mixed sample set according to the policy network to update the policy network and the reward function network, scores each sample in the evaluation data set according to the target reward function network to obtain the corresponding prediction ranking, and then updates the weight coefficient corresponding to each sample in the redistributed mixed sample set according to the prediction ranking to obtain the redistributed weight coefficient, and imitates the redistributed weight coefficient according to the target policy network to obtain the optimized expert sample, which can perform differentiated learning for mixed expert samples of different qualities, improve the sample distribution of the data set, and enhance the generalization ability of the imitation learning agent.

In some embodiments, obtaining the expert sample set includes: performing forward graph reasoning on each sample in the expert sample set based on a visual pre-training model, and determining that a feature map corresponding to a middle layer of the network is a valid feature of the expert sample set.

In this embodiment, the visual pre-trained model is a deep learning model pre-trained using a large-scale dataset, and can generally be used to initialize network parameters for downstream tasks such as image classification, target detection, and image segmentation.

In this embodiment, the general visual reinforcement learning method uses high-dimensional image information as observation of the environment, and has higher requirements on the state abstraction of the feature extractor; because in the adversarial generative imitation learning scenario, factors such as the expressiveness, trainability, and capacity of the image feature extractor are key links in the generalization of the algorithm, generative imitation learning implemented by features extracted from a pre-trained model based on a large-scale visual dataset can improve the generalization ability of the intelligent agent in new tasks or new environments.

Figure 3 is a schematic diagram of the process of extracting the intermediate layer feature map based on the visual pre-training model provided by the present invention. In the embodiment shown in Figure 3, the original feature map is input into the visual pre-training model (corresponding to the convolutional neural network pre-trained with a large-scale visual data set). By fixing the network parameters, only forward reasoning is performed on the input image data, and the feature map of the intermediate layer of the network is extracted as the visual observation input and input into the subsequent imitation learning algorithm.

In this embodiment, during the inference process, only the normalization layer (BatchNorm) of the pre-trained model can be updated by updating the parameters, that is, by keeping the mean and standard deviation parameters corresponding to the normalization layer of the pre-trained model during training, because the statistics updated by BatchNorm help to better adapt to the shift in visual observations, thereby improving the generalization ability of the agent.

The embodiment of the present invention provides an imitation learning mixed sample processing method based on a visual pre-training model. It performs forward graph reasoning on each sample in the expert sample set through the visual pre-training model, and determines that the feature map corresponding to the middle layer of the network is the effective feature of the expert sample set, thereby reducing R&D costs and providing data input for subsequent generative imitation learning algorithms to improve the generalization ability of the intelligent agent in new tasks or new environments.

In some embodiments, the adversarial generative network includes a generative network and a discriminative network; the noise expert samples and the optimal expert samples are processed according to the adversarial generative network to obtain a mixed sample set, including: extracting features from the noise expert samples and the optimal expert samples according to the generative network to obtain a state characterization vector; performing feature discrimination on the source of the state characterization vector according to the discriminative network to obtain a discrimination result; calculating the loss value of the generative network according to the discrimination result and the first loss function, and optimizing the parameters of the generative network according to the loss value of the generative network to obtain an optimized generative network; calculating the loss value of the discriminative network according to the discrimination result and the second loss function, and optimizing the parameters of the discriminative network according to the loss value of the discriminative network to obtain an optimized discriminative network to output a mixed sample set.

In this embodiment, the feature extraction network E (corresponding to the generation network) and the discriminant network D are initialized; the optimal expert sample ξ ^* and the suboptimal expert sample ξ′ in the expert sample set are collected, and the normal distribution noise ∈ is injected into the suboptimal expert sample to obtain the noise expert sample, and then the optimal expert sample and the noise expert sample are input into the feature extraction network E to obtain a new state representation z, that is, a state representation vector; the state representation vector is input into the discriminant network, and the source of the feature is discriminated, and the loss function is calculated according to the discrimination result, and the parameters of the feature extraction network and the discriminant network are optimized.

Among them, the first loss function is determined based on the occupancy measurement of the state-action pair of the optimal expert sample, the occupancy measurement after normal distribution noise, the state representation vector, the conditional probability distribution corresponding to the state representation vector, and the discriminant network parameters; the second loss function is determined based on the occupancy measurement of the state-action pair of the optimal expert sample, the occupancy measurement after normal distribution noise, the state representation vector, the conditional probability corresponding to the state representation vector, and the generation network parameters.

Specifically, the first loss function includes:

The second loss function includes:

Where E is the expectation, D(z) is the discriminant network, ρ ^* (s, a) is the occupancy measure of the optimal expert sample state-action pair, ρ′(s, a) is the occupancy measure after adding normal distribution noise, z is the state representation vector, p(z|x) is the conditional probability distribution corresponding to the state representation vector, θ ^D is the discriminant network parameter, and θ ^E is the generative network parameter.

FIG4 is a third flow chart of the mixed sample processing method for imitation learning based on a visual pre-training model provided by the present invention. In the embodiment shown in FIG4 , noise is injected into the suboptimal expert sample to obtain a noise expert sample, and the optimal expert sample and the noise expert sample are input into the generation network (corresponding to the feature extraction network) to obtain a state representation vector (corresponding to the state feature), and the state representation vector is used to provide mutual information constraints for the optimal expert sample; the state representation vector is input into the discriminant network to perform feature discrimination on the feature source to obtain a discrimination result, and then the loss value of the generation network is calculated according to the discrimination result, and the network gradient of the feature extraction network is updated according to the loss value of the generation network to obtain the optimized generation network; at the same time, the loss value of the discriminant network is calculated according to the discrimination result, and the gradient of the discriminant network is updated according to the loss value of the discriminant network to obtain the optimized discriminant network, thereby outputting a mixed sample set.

In this embodiment, the first loss function also includes: a target regularization term, which is determined according to the KL divergence corresponding to the optimal expert sample and the state representation vector; the second loss function also includes: a target regularization term.

In this embodiment, during the training process of the generative network and the discriminative network, the KL divergence of the optimal expert sample and the extracted features can be used as a regularization term and added to the loss function of the feature extraction network to prevent the extracted features from being too different from the optimal expert sample features, thereby improving the algorithm performance.

The embodiment of the present invention provides an imitation learning mixed sample processing method based on a visual pre-training model, which processes noisy expert samples and optimal expert samples through an adversarial generative network to obtain a mixed sample set, then uses a generative network to extract features from the noisy expert samples and the optimal expert samples to obtain a state representation vector, and uses a discriminant network to perform feature discrimination on the source of the state representation vector to obtain a discrimination result, and finally optimizes the generative network and the discriminant network respectively according to the discrimination result, the first loss function and the second loss function, and outputs a mixed sample set; by introducing noise contrast estimation into adversarial generative imitation learning to improve the data distribution of suboptimal expert samples, simulating the feature difference between the optimal expert samples and the suboptimal expert samples through normal noise, and using an adversarial generation method to train a feature extractor to improve the feature distribution of suboptimal expert samples, thereby improving the performance of the imitation learning algorithm under mixed expert samples.

The following is a description of the mixed sample processing device for imitation learning based on a visual pre-training model provided by the present invention. The mixed sample processing device for imitation learning based on a visual pre-training model described below and the mixed sample processing method for imitation learning based on a visual pre-training model described above can be referenced to each other.

Figure 5 is a structural schematic diagram of the mixed sample processing device for imitation learning based on the visual pre-training model provided by the present invention. As shown in Figure 5, the mixed sample processing device for imitation learning based on the visual pre-training model includes: a sample acquisition module 510, a first processing module 520, a second processing module 530 and a third processing module 540.

The sample acquisition module 510 is used to acquire an expert sample set, where the expert sample set includes an optimal expert sample and a suboptimal expert sample, and each sample in the expert sample set corresponds to a different weight coefficient;

A first processing module 520 is used to add target noise to the suboptimal expert samples to obtain noise expert samples, and process the noise expert samples and the optimal expert samples according to the generative adversarial network to obtain a mixed sample set;

The second processing module 530 is used to calibrate the weight coefficient of each sample in the mixed sample set to obtain a redistributed mixed sample set, predict the redistributed mixed sample set according to the policy network to obtain an action prediction result; score the action prediction result according to the reward function network to obtain a scoring result; train the policy network and the reward function network according to the discriminant loss function and the scoring result to obtain a target policy network and a target reward function network;

The third processing module 540 is used to score each sample in the evaluation data set according to the target reward function network, obtain the predicted ranking corresponding to the evaluation data set, calculate the ranking error loss according to the predicted ranking, update the weight coefficient corresponding to each sample in the redistributed mixed sample set through gradient optimization, obtain the redistributed weight coefficient, and perform imitation learning on the redistributed weight coefficient according to the target strategy network to obtain the optimized expert sample; the evaluation data set belongs to the mixed sample set.

The imitation learning mixed sample processing device based on the visual pre-training model provided by the embodiment of the present invention improves the data distribution of the suboptimal expert samples by adding target noise to the suboptimal expert samples, optimizes the parameters of the adversarial generative network according to the noise expert samples and the optimal expert samples, obtains a mixed sample set, improves the feature distribution of the suboptimal expert samples, and then calibrates the weight coefficient of each sample in the mixed sample set, predicts and scores the calibrated mixed sample set according to the policy network to update the policy network and the reward function network, scores each sample in the evaluation data set according to the target reward function network to obtain the corresponding prediction ranking, and then updates the weight coefficient corresponding to each sample in the redistributed mixed sample set according to the prediction ranking to obtain the redistributed weight coefficient, and imitates the redistributed weight coefficient according to the target policy network to obtain the optimized expert samples, which can perform differentiated learning for mixed expert samples of different qualities, improve the sample distribution of the data set, and enhance the generalization ability of the imitation learning agent.

Figure 6 is a schematic diagram of the structure of the electronic device provided by the present invention. As shown in Figure 6, the electronic device may include: a processor (processor) 610, a communication interface (Communications Interface) 620, a memory (memory) 630 and a communication bus 640, wherein the processor 610, the communication interface 620, and the memory 630 communicate with each other through the communication bus 640. The processor 610 can call the logic instructions in the memory 630 to execute the imitation learning mixed sample processing method based on the visual pre-training model, the method comprising: obtaining an expert sample set, the expert sample set comprising an optimal expert sample and a suboptimal expert sample; adding target noise to the suboptimal expert sample to obtain a noise expert sample, processing the noise expert sample and the optimal expert sample according to the adversarial generation network to obtain a mixed sample set; calibrating the weight coefficient of each sample in the mixed sample set to obtain a redistributed mixed sample set, predicting the redistributed mixed sample set according to the policy network to obtain an action prediction result; scoring the action prediction result according to the reward function network to obtain a scoring result; training the policy network and the reward function network according to the discriminant loss function and the scoring result to obtain a target policy network and a target reward function network; scoring each sample in the evaluation data set according to the target reward function network to obtain a predicted ranking corresponding to the evaluation data set, calculating the ranking error loss according to the predicted ranking, updating the weight coefficient corresponding to each sample in the redistributed mixed sample set through gradient optimization to obtain the redistributed weight coefficient, and imitating learning the redistributed weight coefficient according to the target policy network to obtain an optimized expert sample; the evaluation data set belongs to the mixed sample set.

In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the imitation learning mixed sample processing method based on the visual pre-training model provided by the above methods, and the method includes: obtaining an expert sample set, the expert sample set includes an optimal expert sample and a suboptimal expert sample; adding target noise to the suboptimal expert sample to obtain a noise expert sample, processing the noise expert sample and the optimal expert sample according to a generative adversarial network to obtain a mixed sample set; calibrating the weight coefficient of each sample in the mixed sample set to obtain a redistributed mixed sample set, and Predict the redistributed mixed sample set to obtain the action prediction result; score the action prediction result according to the reward function network to obtain the scoring result; train the policy network and the reward function network according to the discriminant loss function and the scoring result to obtain the target policy network and the target reward function network; score each sample in the evaluation data set according to the target reward function network to obtain the predicted ranking corresponding to the evaluation data set, calculate the ranking error loss according to the predicted ranking, update the weight coefficient corresponding to each sample in the redistributed mixed sample set through gradient optimization to obtain the redistributed weight coefficient, and imitate the redistributed weight coefficient according to the target policy network to obtain the optimized expert sample; the evaluation data set belongs to the mixed sample set.

On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the mixed sample processing method for imitation learning based on the visual pre-training model provided by the above methods, the method comprising: obtaining an expert sample set, the expert sample set comprising an optimal expert sample and a suboptimal expert sample; adding target noise to the suboptimal expert sample to obtain a noise expert sample, processing the noise expert sample and the optimal expert sample according to a generative adversarial network to obtain a mixed sample set; calibrating a weight coefficient for each sample in the mixed sample set to obtain a redistributed mixed sample set, predicting the redistributed mixed sample set according to a strategy network, Obtain action prediction results; score the action prediction results according to the reward function network to obtain scoring results; train the policy network and the reward function network according to the discriminant loss function and the scoring results to obtain the target policy network and the target reward function network; score each sample in the evaluation data set according to the target reward function network to obtain the predicted ranking corresponding to the evaluation data set, calculate the ranking error loss according to the predicted ranking, update the weight coefficient corresponding to each sample in the redistributed mixed sample set through gradient optimization, obtain the redistributed weight coefficient, perform imitation learning on the redistributed weight coefficient according to the target policy network to obtain the optimized expert sample; the evaluation data set belongs to a mixed sample set.

The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM/RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (9)

1. A method for processing a simulated learning mixture based on a vision pre-training model, comprising:

acquiring an expert sample set, wherein the expert sample set comprises an optimal expert sample and a suboptimal expert sample;

Adding target noise to the suboptimal expert sample to obtain a noise expert sample, and processing the noise expert sample and the optimal expert sample according to an countermeasure generation network to obtain a mixed sample set;

calibrating weight coefficients for each sample in the mixed sample set to obtain a redistributed mixed sample set, and predicting the redistributed mixed sample set according to a strategy network to obtain an action prediction result; scoring the action prediction result according to a reward function network to obtain a scoring result; training the strategy network and the reward function network according to the discrimination loss function and the scoring result to obtain a target strategy network and a target reward function network;

Scoring each sample in an evaluation data set according to the target reward function network to obtain a prediction ranking corresponding to the evaluation data set, calculating ranking error loss according to the prediction ranking, updating weight coefficients corresponding to each sample in the redistributed mixed sample set through gradient optimization to obtain redistributed weight coefficients, and performing simulated learning on the redistributed weight coefficients according to the target strategy network to obtain optimized expert samples; the evaluation dataset belongs to the mixed sample set; the acquiring the expert sample set includes:

Performing forward graph reasoning on each sample in the expert sample set based on a vision pre-training model, and determining a feature graph corresponding to a network middle layer as effective features of the expert sample set;

The forward graph reasoning on each sample in the expert sample set based on the vision pre-training model comprises:

Inputting original feature graphs corresponding to all samples in the expert sample set into the vision pre-training model, and only forward reasoning is carried out on input image data through fixed network parameters, and in the reasoning process, only mean value and standard deviation parameters corresponding to a normalization layer of the vision pre-training model are updated;

The expert sample set includes training samples or test samples required for at least one of the following scenarios: image classification, object detection and image segmentation.

2. The visual pre-training model-based simulated learning mixture sample processing method of claim 1, wherein the countermeasure generation network comprises a generation network and a discrimination network;

The processing the noise expert sample and the optimal expert sample according to the countermeasure generation network, and obtaining a mixed sample set includes:

extracting characteristic features of the noise expert samples and the optimal expert samples according to the generating network to obtain a state characterization vector;

performing feature discrimination on the source of the state characterization vector according to the discrimination network to obtain discrimination results;

Calculating a loss value of the generating network according to the judging result and the first loss function, and carrying out parameter optimization on the generating network according to the loss value of the generating network to obtain the optimized generating network; calculating a loss value of the discrimination network according to the discrimination result and the second loss function, and carrying out parameter optimization on the discrimination network according to the loss value of the discrimination network to obtain the optimized discrimination network so as to output the mixed sample set;

The first loss function is determined based on an occupancy metric of an optimal expert sample state action pair, an occupancy metric after normal distribution noise, the state characterization vector, a conditional probability distribution corresponding to the state characterization vector, and a discrimination network parameter, and the second loss function is determined based on an occupancy metric of the optimal expert sample state action pair, an occupancy metric after normal distribution noise, the state characterization vector, a conditional probability corresponding to the state characterization vector, and a generation network parameter.

3. The visual pre-training model-based simulated learning mixture sample processing method of claim 2, wherein the first loss function comprises:

the second loss function includes:

Where E is the expectation, D (z) is the discrimination network, ρ ^* (s, a) is the occupancy measure of the optimal expert sample state action pair, ρ' (s, a) is the occupancy measure after adding normal distribution noise, z is the state characterization vector, p (z|x) is the conditional probability distribution corresponding to the state characterization vector, θ ^D is the discrimination network parameter, and θ ^E is the generation network parameter.

4. The visual pre-training model-based simulated learning mixture sample processing method of claim 2, wherein the first loss function further comprises:

the target regular term is determined according to the KL divergence corresponding to the optimal expert sample and the state characterization vector;

The second loss function further includes: the target regularization term.

5. The visual pre-training model-based simulated learning hybrid sample processing method of claim 1, wherein said evaluation dataset is obtained by said expert sample set in a priori ordering;

Calculating the sorting error loss according to the prediction sorting, updating the weight coefficient corresponding to each sample in the redistributed mixed sample set through gradient optimization, wherein the obtaining the redistributed weight coefficient comprises the following steps:

calculating a ranking difference penalty by the predictive ranking, the prior ranking, and a ranking penalty function;

Determining the weight coefficient after redistribution according to the sorting difference loss;

the ordering loss function includes:

wherein i and j are sample numbers, The expected return is accumulated for the true number i sample,Accumulating expected returns for predictions for sample i; the expected return is accumulated for the true sample j, The expected return is accumulated for the prediction for sample j.

6. An imitation learning hybrid sample processing device based on a vision pre-training model, comprising:

the sample acquisition module is used for acquiring an expert sample set, wherein the expert sample set comprises an optimal expert sample and a suboptimal expert sample, and each sample in the expert sample set corresponds to different weight coefficients;

The first processing module is used for adding target noise to the suboptimal expert sample to obtain a noise expert sample, and processing the noise expert sample and the optimal expert sample according to an antagonism generation network to obtain a mixed sample set;

the second processing module is used for calibrating weight coefficients for each sample in the mixed sample set to obtain a redistributed mixed sample set, and predicting the redistributed mixed sample set according to a strategy network to obtain an action prediction result; scoring the action prediction result according to a reward function network to obtain a scoring result; training the strategy network and the reward function network according to the discrimination loss function and the scoring result to obtain a target strategy network and a target reward function network;

The third processing module is used for scoring each sample in the evaluation data set according to the target reward function network to obtain a prediction ranking corresponding to the evaluation data set, calculating ranking error loss according to the prediction ranking, updating weight coefficients corresponding to each sample in the redistributed mixed sample set through gradient optimization to obtain redistributed weight coefficients, and performing imitation learning on the redistributed weight coefficients according to the target strategy network to obtain optimized expert samples; the evaluation dataset belongs to the mixed sample set;

The sample acquisition module is specifically configured to:

Performing forward graph reasoning on each sample in the expert sample set based on a vision pre-training model, and determining a feature graph corresponding to a network middle layer as effective features of the expert sample set;

The forward graph reasoning on each sample in the expert sample set based on the vision pre-training model comprises:

Inputting original feature graphs corresponding to all samples in the expert sample set into the vision pre-training model, and only forward reasoning is carried out on input image data through fixed network parameters, and in the reasoning process, only mean value and standard deviation parameters corresponding to a normalization layer of the vision pre-training model are updated;

The expert sample set includes training samples or test samples required for at least one of the following scenarios: image classification, object detection and image segmentation.

7. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the visual pre-training model-based simulated learning hybrid sample processing method of any of claims 1 to 5 when the program is executed by the processor.

8. A non-transitory computer readable storage medium having stored thereon a computer program, which when executed by a processor implements a method of simulated learning mixed sample processing based on a visual pre-training model as claimed in any of claims 1 to 5.

9. A computer program product comprising a computer program which, when executed by a processor, implements a method of simulated learning hybrid sample processing based on a visual pre-training model as claimed in any one of claims 1 to 5.