CN110009044A - A kind of model training method and device, image processing method and device - Google Patents
A kind of model training method and device, image processing method and device Download PDFInfo
- Publication number
- CN110009044A CN110009044A CN201910280851.9A CN201910280851A CN110009044A CN 110009044 A CN110009044 A CN 110009044A CN 201910280851 A CN201910280851 A CN 201910280851A CN 110009044 A CN110009044 A CN 110009044A
- Authority
- CN
- China
- Prior art keywords
- image
- training
- model
- feature
- training image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Image Analysis (AREA)
Abstract
The embodiment of the present application discloses a kind of model training method and device, image processing method and device, since the image procossing with fisrt feature can be become the image with second feature by the first generation model, second, which generates model, can become the image with fisrt feature for the image procossing with second feature, if the image restoring crossed by the first generation model treatment can be original image by the second generation model, the image restoring crossed by the second generation model treatment can be original image by the first generation model, it may be considered that the first generation model and the second generation model are more accurate to the specific aim processing of image, based on this, first object function can be minimized for the training setting of image processing model and/or minimize the second bound for objective function, image processing model is updated using model optimization algorithm, The image processing model enable is accurate to handle image, and the picture quality of output is also higher, improves user experience.
Description
Technical field
This application involves computer field more particularly to a kind of model training methods and device, image processing method and dress
It sets.
Background technique
During image processing, it will usually have the demand handled the target position in image, such as removal people
Glasses, beard etc. as in, in another example the facial expression etc. of adjustment personage, in another example increasing glasses, beard etc. for portrait.In order to
Meet this demand, image processing model can be enable to carry out image specific by being trained to image processing model
Processing.However, for loss of detail than more serious, picture quality is poor in the output image of existing image processing model, it cannot
It meets the needs of users.
Summary of the invention
In order to solve the problems, such as that the picture quality for exporting image in prior art machine learning model is poor, the application is implemented
Example provides a kind of model training method and device, image processing method and device.
The embodiment of the present application provides a kind of model training method, comprising:
Training image is obtained, the training image includes the first training image and/or the second training image, wherein described the
One training image is the image for including fisrt feature, and second training image is the image for including second feature, described first
Feature and the second feature are for same attribute and with the feature of different presentations；
Image processing model is trained using the training image, wherein described image processing model includes first raw
Model is generated at model and second；
Training process includes:
First training image input the first generation model is obtained into third training image, by third training
Image input the second generation model obtains the 4th training image, the third training image be include the second feature
Image, the 4th training image is the image for including the fisrt feature；
Second training image input the second generation model is obtained into the 5th training image, by the 5th training
Image input the first generation model obtains the 6th training image, the 5th training image be include the fisrt feature
Image, the 6th training image is the image for including the second feature；
The training process further includes following constraint condition: first object function minimization and/or the second objective function are most
Smallization；
The first object function is used to express the difference between the 4th training image and first training image,
Second objective function is used to express the difference of the 6th training image and second training image.
Optionally, the first object function
laid 0=| | x0-G1(G0(x0))||1
Wherein, x0Indicate first training image, G0Indicate that described first generates model, G0(x0) indicate the third
Training image, G1Indicate that described second generates model, G1(G0(x0)) indicate the 4th training image, | | x0-G1(G0(x0))|
|1It indicates to x0-G1(G0(x0)) calculate L1 norm.
Optionally, second objective function
laid 1=| | x1-G0(G1(x1))||1
Wherein, x1Indicate second training image, G1Indicate that described second generates model, G1(x1) indicate the described 5th
Training image, G0Indicate that described first generates model, G0(G1(x1)) indicate the 6th training image, | | x1-G0(G1(x1))|
|1It indicates to x1-G0(G1(x1)) calculate L1 norm.
Optionally, the fisrt feature is that target object has hot spot on the face, and the second feature is that target object does not have on the face
There is hot spot.
Optionally, the fisrt feature is that target object is worn glasses on the face, the second feature be target object on the face not
It wears glasses.
The embodiment of the present application also provides a kind of model training apparatus, described device includes:
Training image acquiring unit, for obtaining training image, the training image includes the first training image and/or the
Two training images, wherein first training image is the image for including fisrt feature, second training image be include
The image of two features, the fisrt feature and the second feature are for same attribute and with the feature of different presentations；
Model training unit, for being trained using the training image to image processing model, wherein described image
Handling model includes that the first generation model and second generate model；
Training process includes:
First training image input the first generation model is obtained into third training image, by third training
Image input the second generation model obtains the 4th training image, the third training image be include the second feature
Image, the 4th training image is the image for including the fisrt feature；
Second training image input the second generation model is obtained into the 5th training image, by the 5th training
Image input the first generation model obtains the 6th training image, the 5th training image be include the fisrt feature
Image, the 6th training image is the image for including the second feature；
The training process further includes following constraint condition: first object function minimization and/or the second objective function are most
Smallization；
The first object function is used to express the difference between the 4th training image and first training image,
Second objective function is used to express the difference of the 6th training image and second training image.
Optionally, the first object function
laid 0=| | x0-G1(G0(x0))||1
Wherein, x0Indicate first training image, G0Indicate that described first generates model, G0(x0) indicate the third
Training image, G1Indicate that described second generates model, G1(G0(x0)) indicate the 4th training image, | | x0-G1(G0(x0))|
|1It indicates to x0-G1(G0(x0)) calculate L1 norm.
Optionally, second objective function
laid 1=| | x1-G0(G1(x1))||1
Wherein, x1Indicate second training image, G1Indicate that described second generates model, G1(x1) indicate the described 5th
Training image, G0Indicate that described first generates model, G0(G1(x1)) indicate the 6th training image, | | x1-G0(G1(x1))|
|1It indicates to x1-G0(G1(x1)) calculate L1 norm.
Optionally, the fisrt feature is that target object has hot spot on the face, and the second feature is that target object does not have on the face
There is hot spot.
Optionally, the fisrt feature is that target object is worn glasses on the face, the second feature be target object on the face not
It wears glasses.
The embodiment of the present application also provides a kind of image processing methods, which comprises
The first image is obtained, the first image includes fisrt feature；
The first image is input in the first generation model, the second image is obtained, second image includes second
Feature, the fisrt feature and the second feature are for same attribute and raw with the feature of different presentations, described first
A kind of model training method such as provided by the embodiments of the present application is used to be trained to obtain at model.
The embodiment of the present application also provides a kind of image processing apparatus, described device includes:
First image acquisition unit, for obtaining the first image, the first image includes fisrt feature；
Second image acquisition unit obtains the second image for the first image to be input in the first generation model,
Second image includes second feature, and the fisrt feature and the second feature, which are for same attribute and with difference, is in
Existing feature, the first generation model use a kind of model training method such as provided by the embodiments of the present application to be trained
It arrives.
The embodiment of the present application also provides another image processing methods, which comprises
Third image is obtained, the third image includes second feature；
The third image is input in the second generation model, the 4th image is obtained, described image includes fisrt feature,
The fisrt feature and the second feature are to generate model for same attribute and with the feature of different presentations, described second
It is trained to obtain using a kind of model training method such as provided by the embodiments of the present application.
The embodiment of the present application also provides another image processing apparatus, described device includes:
Third image acquisition unit, for obtaining third image, the third image includes second feature；
4th image acquisition unit obtains the 4th image for the third image to be input in the second generation model,
Described image includes fisrt feature, and the fisrt feature and the second feature are for same attribute and with different presentations
Feature, the second generation model use a kind of model training method such as provided by the embodiments of the present application to be trained to obtain.
The embodiment of the present application provides a kind of model training method and device, can first obtain training image, training image
Including the first training image and/or the second training image, wherein the first training image is the image for including fisrt feature, the second instruction
Practicing image is the image for including second feature, and fisrt feature and second feature are for same attribute and with the spy of different presentations
Sign, such as wearing glasses and do not wear glasses can be respectively as fisrt feature and second feature；Then using training image to image
Processing model is trained, and wherein image processing model includes that the first generation model and second generate model, wherein by the first instruction
Practice image and be input to the available third training image with second feature of the first generation model, third training image is inputted
Model, available the 4th training image with fisrt feature, correspondingly, the second training image is input to are generated to second
Second generates available the 5th training image with fisrt feature of model, and the 5th training image is input to the first generation mould
Type, available the 6th training image with second feature can pass through first object function table in the embodiment of the present application
Up to the difference of the first training image and the 4th training image, the second training image and the 6th training are expressed by the second objective function
The difference of image will minimize first object function and/or minimize the second objective function as training image processing model
Constraint condition, the image processing model after being trained.
Since the image procossing with fisrt feature can be become the image with second feature by the first generation model, the
Two, which generate model, can become the image with fisrt feature for the image procossing with second feature, if the second generation model can
Will pass through the image restoring that the first generation model treatment is crossed for original image, first, which generates model, can will pass through the second life
It is original image at the image restoring that model treatment is crossed, it may be considered that first generates model and the second generation model to image
Specific aim processing it is more accurate, be based on this, can for image processing model training setting minimize first object function and/
Or the second bound for objective function is minimized, image processing model, the image made are updated using model optimization algorithm
Processing model accurate can be handled image, and the picture quality of output is also higher, improve user experience.
Detailed description of the invention
In order to illustrate the technical solutions in the embodiments of the present application or in the prior art more clearly, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
The some embodiments recorded in application, for those of ordinary skill in the art, without creative efforts,
It can also be obtained according to these attached drawings other attached drawings.
Fig. 1 is a kind of flow chart of model training method provided by the embodiments of the present application；
Fig. 2 is a kind of image procossing schematic diagram of image processing model provided by the embodiments of the present application；
Fig. 3 is the image procossing schematic diagram of another image processing model provided by the embodiments of the present application；
Fig. 4 is the image procossing schematic diagram of another image processing model provided by the embodiments of the present application；
Fig. 5 is a kind of flow chart of image processing method provided by the embodiments of the present application；
Fig. 6 is the flow chart of another image processing method provided by the embodiments of the present application；
Fig. 7 is a kind of structural block diagram of model training apparatus provided by the embodiments of the present application；
Fig. 8 is a kind of structural block diagram of image processing apparatus provided by the embodiments of the present application；
Fig. 9 is the structural block diagram of another image processing apparatus provided by the embodiments of the present application.
Specific embodiment
In order to make those skilled in the art more fully understand application scheme, below in conjunction in the embodiment of the present application
Attached drawing, the technical scheme in the embodiment of the application is clearly and completely described, it is clear that described embodiment is only this
Apply for a part of the embodiment, instead of all the embodiments.Based on the embodiment in the application, those of ordinary skill in the art exist
Every other embodiment obtained under the premise of creative work is not made, shall fall in the protection scope of this application.
In the prior art, image procossing can be made by being trained using model optimization algorithm to image processing model
Model can be handled the target position in image, such as glasses, beard in removal portrait etc., or adjustment task
Facial expression etc., or glasses, beard etc. are increased to portrait.However in the output image of existing image processing model, details
Than more serious, picture quality is poor for loss, cannot meet the needs of users, such as during removing the glasses in portrait,
Other positions that portrait may be also resulted in generate variation, and the portrait before the portrait that makes that treated and processing is not " as ".
It for example, include glasses in facial image in some scenes, and glasses reflection generates hot spot, if using the people
Face image carries out Retina transplantation, then the hot spot of glasses reflection can impact analysis result, therefore can first pass through image
The hot spot for handling model removal glasses reflection, further according to treated, image carries out Retina transplantation, to improve Retina transplantation result
Accuracy.However, not removing only the hot spot of glasses reflection in the output image of existing image processing model, can also lead
It causes the other positions of facial image to change, such as the variation such as big small distance of eye shape, equally will affect Retina transplantation knot
The accuracy of fruit.
Based on this, the embodiment of the present application provides a kind of model training method and device, can first obtain training image, instructs
Practicing image includes the first training image and/or the second training image, wherein the first training image is the image for including fisrt feature,
Second training image is the image for including second feature, and fisrt feature and second feature, which are for same attribute and with difference, is in
Existing feature, such as wearing glasses and do not wear glasses can be respectively as fisrt feature and second feature；Then training image is utilized
Image processing model is trained, wherein image processing model includes that the first generation model and second generate model, wherein will
First training image is input to the available third training image with second feature of the first generation model, and third training is schemed
As being input to the second generation model, available the 4th training image with fisrt feature, correspondingly, by the second training image
It is input to available the 5th training image with fisrt feature of the second generation model, the 5th training image is input to first
Model is generated, available the 6th training image with second feature can pass through first object in the embodiment of the present application
The difference of function representation the first training image and the 4th training image expresses the second training image and the by the second objective function
The difference of six training images will minimize first object function and/or minimize the second objective function as training image processing
The constraint condition of model, the image processing model after being trained.
Since the image procossing with fisrt feature can be become the image with second feature by the first generation model, the
Two, which generate model, can become the image with fisrt feature for the image procossing with second feature, if the second generation model can
Will pass through the image restoring that the first generation model treatment is crossed for original image, first, which generates model, can will pass through the second life
It is original image at the image restoring that model treatment is crossed, it may be considered that first generates model and the second generation model to image
Specific aim processing it is more accurate, be based on this, can for image processing model training setting minimize first object function and/
Or the second bound for objective function is minimized, image processing model, the image made are updated using model optimization algorithm
Processing model accurate can be handled image, and the picture quality of output is also higher, improve user experience.
Embodiment one:
Referring to Fig. 1, which is a kind of flow chart of model training method provided by the embodiments of the present application.The present embodiment provides
Model training method include the following steps:
S101 obtains training image.
Training image is the image for training image processing model, and image processing model is for image
Therefore the model of reason can targetedly train image processing model, be made according to reality to the processing needs of image
Image processing model after training has corresponding function, equally, training image and root for training image processing model
Factually border needs to select to the processing of image.For example, image processing model can be handled the fisrt feature in image
As second feature, then the training image for training image processing model can be include the image of fisrt feature and including the
The image of two features.For example, the function of training pattern is to reject glasses, then training image is the data set of non-wearing spectacles.
In the embodiment of the present application, image processing model can be unsupervised approaches model, such as be fought based on production
The model of network (Generative Adversarial Networks, GAN), the training figure for training image processing model
As that can not need manual identified and increase label.
Training image can only include the first training image, can also only include the second training image, can also both include
First training image, and including the second training image.Wherein, the first training image is the image for including fisrt feature, the second instruction
Practicing image is the image for including second feature, and fisrt feature and second feature are for same attribute and with the spy of different presentations
Sign.
For example, it if attribute is glasses, wears glasses on the face then fisrt feature can be target object, second feature can
To be that target object is not worn glasses on the face；If attribute is hot spot, then fisrt feature can be target object hot spot on the face, the
Two features can be target object on the face without hot spot；If attribute is beard, then fisrt feature can be target object on the face
There is beard, second feature can be target object on the face without beard；If attribute is mouth, then fisrt feature can be target pair
As grin, second feature can be target object not grin etc..Certainly, fisrt feature and second feature can be interchanged,
I.e. fisrt feature can be target object and not wear glasses on the face, and second feature can be target object and wear glasses on the face.
In the scene of hot spot for removing glasses reflection by image processing model, fisrt feature can be target object face
On have glasses, and there are hot spot on glasses, second feature, which can be, glasses on target object, and hot spot is not present on glasses,
It certainly, is that target object has glasses on the face, and there are when hot spot on glasses, corresponding second feature is also possible in fisrt feature
On the face without glasses, the image with both second feature may be incorporated for instructing image processing model target object
Practice.
S102 is trained image processing model using training image.
In the embodiment of the present application, image processing model may include that the first generation model and second generate model.Wherein,
First training image is input in the first generation model, the available third training image including second feature, that is,
It says, the first generation model has the ability that fisrt feature is treated as to second feature；And the second training image is input to
Two generate in models, available the 5th training image including fisrt feature, that is to say, that second generates model with by the
Two characteristic processings become the ability of fisrt feature.
For theoretically, the first generation model and the second generation model are antipodal processes to the processing of image, the
The 4th training image that the third training image of one generation model output is obtained by the second generation model treatment, should be with first
Training image is identical, and the 6th instruction that the 5th training image of the second generation model output is obtained by the first generation model treatment
Practice image, it should be identical as the second training image.
However in fact, image processing process is not up to such as above-mentioned perfect condition.Therefore, in the embodiment of the present application
In, the second target can be passed through by the difference between the 4th training image of first object function representation and the first training image
Difference between the 6th training image of function representation and the second training image can in the training process to image processing model
Will minimize first object function and/or minimize the second objective function as the constraint condition in training process, that is,
The difference of the 4th training image and the first training image is reduced in the training process, and/or, reduce the 6th training image and second
The difference of training image, to make image processing model close to perfect condition.It is understood that the function of image processing model
Closer to perfect condition, the accuracy of image procossing is also higher, for other features except fisrt feature or second feature
Variation is just smaller.
A kind of image procossing schematic diagram of image processing model is shown with reference to Fig. 2, wherein by the first training image x0It is defeated
Enter to first and generates model G0, obtain third training image G0(x0), by third training image G0(x0) it is input to the second generation mould
Type G1, obtain the 4th training image G1(G0(x0)), then first object function can indicate are as follows:
laid 0=| | x0-G1(G0(x0))||1 (1)
Wherein, | | x0-G1(G0(x0))||1It indicates to x0-G1(G0(x0)) calculate L1 norm.
First object function is minimized, can with specifically, initialisation image processing model weight parameter, setting
The hyper parameter of model generates model and second by gradient descent method adjustment first and generates the weight parameter of model, makes the first mesh
Scalar functions minimize.Hyper parameter may include exercise wheel number n, learning rate lr, batch processing population size bn etc..
The image procossing schematic diagram of another image processing model is shown with reference to Fig. 3, wherein by the first training image x1
It is input to the second generation model G1, obtain the 5th training image G1(x1), by the 5th training image G1(x1) it is input to the first generation
Model G0, obtain the 6th training image G0(G1(x1)), then the second objective function can indicate are as follows:
laid 1=| | x1-G0(G1(x1))||1 (2)
Wherein, | | x1-G0(G1(x1))||1It indicates to x1-G0(G1(x1)) calculate L1 norm.
Second objective function is minimized, can with specifically, initialisation image processing model weight parameter, setting
The hyper parameter of model generates model and second by gradient descent method adjustment first and generates the weight parameter of model, makes the second mesh
Scalar functions minimize.Hyper parameter may include exercise wheel number n, learning rate lr, batch processing population size bn etc..
First object function will be minimized and/or minimize the second objective function as the constraint condition in training process,
It can use model optimization algorithm and update image processing model, enable updated image processing model more accurately to image
It is handled.Wherein, using the parameter in the adjustable image processing model of model optimization algorithm, thus to image processing model
It is updated.
In the embodiment of the present application, have since the first generation model can become the image procossing with fisrt feature
The image of second feature, second, which generates model, can become the figure with fisrt feature for the image procossing with second feature
Picture, if the image restoring crossed by the first generation model treatment can be original image by the second generation model, first is generated
The image restoring crossed by the second generation model treatment can be original image by model, it may be considered that first generates model
It is more accurate with specific aim processing of the second generation model to image, it is based on this, can be arranged for the training of image processing model
It minimizes first object function and/or minimizes the second bound for objective function, utilize model optimization algorithm more new images
Model is handled, the image processing model enable is accurate to handle image, and the picture quality of output is also higher,
Improve user experience.
Method based on the model training that above embodiments provide, can also have in the embodiment of the present application other to constrain item
Part, specifically, at least one with minor function can also be minimized: the loss function of discrimination model, first generate the of model
One loss function, first generate the second loss function of model, the third loss function of the second generation model, the second generation model
The 4th loss function.
Specifically, showing with reference to Fig. 4 image procossing for showing another image processing model provided by the embodiments of the present application
It is intended to, which includes the first generation model, the second generation model and discrimination model, and wherein discrimination model is for sentencing
The image of disconnected input is after probability and the input picture of original image generate model or the second generation model treatment for first
The probability of image differentiates the result is that probability value.The loss function of discrimination model can indicate are as follows:
ld=lcls(p, t)=- log (pt), t=0,1,2 (3)
Wherein, ptIndicate that image belongs to the probability of t class, specific 0 class indicates the first input picture for generating model, 1 class
Indicate the second input picture for generating model, 2 classes indicate the first output image for generating model or the second generation model.
In the embodiment of the present application, the loss function for minimizing discrimination model can also be used as instruction to image processing model
Constraint condition during white silk, to reduce discrimination model to the differentiation difference of original image and treated image.
When it is implemented, by the first training image x0It is input to the first generation model G0, obtain third training imageThe
Three training images can also be expressed as G0(x0), it will be in the first training image and third training image input discrimination model D；By
Two training image x1It is input to the second generation model G1In, obtain the 5th training image5th training image can also indicate
For G1(x1), the second training image and the 5th training image are inputted in discrimination model D, to make discrimination model to the first training
Image x0, the second training image x1, third training image5th training imageDifferentiated, wherein the first training image
Probability for 0 class is larger, and the second training image is that the probability of 1 class is larger, and third training image and the 5th training image are 2 classes
Probability is larger.
Further, it is also possible to by third training imageIt is input to the second generation model G1, obtain the 4th training image G1(G0
(x0)), by the 5th training imageIt is input to the first generation model G0, obtain the 6th training image G0(G1(x1))。
The first-loss function l of first generation model as a result,g 01It can indicate are as follows:
lg 01=γ laid 0+lo 01=γ laid 0+lGAN 0+αlpix 0+βlper 0 (4)
Wherein, lo 01The third objective function for generating model for first, lGAN 0Indicate the first GAN loss for generating model,
lpix 0Indicate the first residual error regularization value for generating model, lper 0Indicate the first perception loss for generating model, α, β and γ divide
Not Wei loss function weight parameter, those skilled in the art can rule of thumb set, laid 0It can be with reference formula (1).Tool
Body,
lGAN 0=-log (D (G0(x0))), (5)
lpix 0=| | r0||1, (6)
Wherein, r0For x0It is input to the first generation model G0Obtained in residual image,KnowIt respectively indicates
x0WithAccording to it is certain rule mapping after value.
First generates the second loss function l of modelg 02It can indicate are as follows:
lg 02=γ laid 0+lo 02=γ laid 0+lGAN 0+ldual 0+αlpix 0+βlper 0 (8)
Wherein, lo 02The 4th objective function for generating model for first, lGAN 0Indicate the first GAN loss for generating model,
ldual 0Indicate first loss function during the paired-associate learning that first generates model and the second generation model, lpix 0Indicate the
One generates the residual error regularization value of model, lper 0Indicate the first perception loss for generating model, α, β and γ are respectively to lose letter
Several weight parameters, those skilled in the art can rule of thumb set, lGAN 0、lpix 0、lper 0Can with reference formula (5),
(6) and (7), laid 0It can be with reference formula (1).Specifically,
In the embodiment of the present application, the first first-loss function and/or the second loss function for generating model is minimized,
It can be used as the constraint condition in the training process to image processing model.
Correspondingly, second generates the third loss function l of modelg 11It can indicate are as follows:
lg 11=γ laid 1+lo 11=γ laid 1+lGAN 1+αlpix 1+βlper 1 (10)
Wherein, lo 11The 5th objective function for generating model for second, lGAN 1Indicate the second GAN loss for generating model,
lpix 1Indicate the second residual error regularization value for generating model, lper 1Indicate the second perception loss for generating model, α, β and γ divide
Not Wei loss function weight parameter, those skilled in the art can rule of thumb set, laid 1It can be with reference formula (2).Tool
Body,
lGAN 1=-log (1-D (G1(x1))), (11)
lpix 1=| | r1||1, (12)
Wherein, wherein r1For x1It is input to the second generation model G1Obtained in residual image,WithRespectively
Indicate x1WithAccording to it is certain rule mapping after value.
Second generates the 4th loss function l of modelg 12It can indicate are as follows:
lg 12=γ laid 1+lo 12=γ laid 1+lGAN 1+ldual 1+αlpix 1+βlper 1 (14)
Wherein, lo 12The 5th objective function for generating model for second, lGAN 1Indicate the second GAN loss for generating model,
ldual 1Indicate second loss function during the paired-associate learning that first generates model and the second generation model, lpix 1Indicate the
Two generate the residual error regularization value of model, lper 1Indicate the second perception loss for generating model, α, β and γ are respectively to lose letter
Several weight parameters, those skilled in the art can rule of thumb set, lGAN 1、lpix 1、lper 1Can with reference formula (11),
(12) and (13), laid 1It can be with reference formula (2).Specifically,
In the embodiment of the present application, the second third loss function and/or the 4th loss function for generating model is minimized,
It can be used as the constraint condition in the training process to image processing model.
Embodiment two
Based on a kind of model training method that above embodiments provide, refering to what is shown in Fig. 5, the embodiment of the present application also provides
A kind of image processing method is handled the first image using the image processing model that training is completed, specifically, may include
Following steps:
S201 obtains the first image.
The image processing model that training is completed is can to carry out image procossing for a certain attribute of image, therefore input
First image of image processing model may include fisrt feature, and fisrt feature can be with the explanation in reference implementation example one, herein
It does not repeat them here.
In the scene of hot spot for removing glasses reflection by image processing model, fisrt feature can be target object face
On have glasses, and there are hot spots on glasses.
First image is input in the first generation model, obtains the second image by S202.
First generation model is to be trained using the model training method of the embodiment of the present application one, due to first
Second feature can be treated as fisrt feature by generating model, so the second image includes second feature, fisrt feature and the
Two features are for same attribute and with the feature of different presentations.
For example, if attribute is glasses, fisrt feature is that target object is worn glasses on the face, then second feature can be
Target object is not worn glasses on the face；If attribute is hot spot, fisrt feature is that target object has hot spot on the face, then second feature can
To be target object on the face without hot spot；If attribute is beard, fisrt feature is that target object has beard on the face, then second is special
Sign can be target object on the face without beard；If attribute is mouth, fisrt feature is target object grin, then second feature
It can be target object not grin etc..
In the scene of hot spot for removing glasses reflection by image processing model, fisrt feature can be target object face
On have glasses, and there are hot spot on glasses, second feature, which can be, glasses on target object, and hot spot is not present on glasses,
It certainly, is that target object has glasses on the face, and there are when hot spot on glasses, corresponding second feature is also possible in fisrt feature
Target object is on the face without glasses.
As first generation model be according to the embodiment of the present application one provide model training method training made of, can
Targetedly the first image is handled, the accuracy of image procossing is higher, and the picture quality of output is also higher, therefore improves
User experience.
Embodiment three
Based on a kind of model training method that above embodiments provide, refering to what is shown in Fig. 6, the embodiment of the present application also provides
Another image processing method is handled third image using the image processing model that training is completed, specifically, can wrap
Include following steps:
S301 obtains third image.
The image processing model that training is completed is can to carry out image procossing for a certain attribute of image, therefore input
The third image of image processing model may include second feature.Second feature can be with the explanation in reference implementation example one, herein
It does not repeat them here.
Third image is input in the second generation model, obtains the 4th image by S302.
Second generation model is to be trained using the model training method of the embodiment of the present application one, due to second
Fisrt feature can be treated as second feature by generating model, so the 4th image includes fisrt feature, fisrt feature and the
Two features are for same attribute and with the feature of different presentations.
For example, if attribute is glasses, second feature is that target object is not worn glasses on the face, then fisrt feature can be with
It is that target object is worn glasses on the face；If attribute is hot spot, second feature is target object on the face without hot spot, then fisrt feature
Can be target object has hot spot on the face；If attribute is beard, second feature is target object on the face without beard, then first
Feature can be target object beard on the face；If attribute is mouth, second feature can be target object not grin, then the
One feature can be target object grin etc..
As second generation model be according to the embodiment of the present application one provide model training method training made of, can
Targetedly third image is handled, the accuracy of image procossing is higher, and the picture quality of output is also higher, therefore improves
User experience.
Based on a kind of model training method that above embodiments provide, the embodiment of the present application also provides a kind of model trainings
Its working principle is described in detail with reference to the accompanying drawing in device.
Example IV
Referring to Fig. 7, which is a kind of structural block diagram for model training apparatus that the embodiment of the present application four provides.The present embodiment
The model training apparatus of offer includes:
Training image acquiring unit 110, for obtaining training image, the training image include the first training image and/
Or second training image, wherein first training image is the image for including fisrt feature, second training image is packet
The image of second feature is included, the fisrt feature and the second feature are for same attribute and with the spy of different presentations
Sign；
Model training unit 120, for being trained using the training image to image processing model, wherein the figure
As processing model includes that the first generation model and second generate model；
Training process includes following constraint condition: first object function minimization and/or the second the minimization of object function；
First training image is input to the first generation model for expression and obtained by the first object function
Third training image be input to obtained the 4th training image of the second generation model again, with first training image it
Between difference, the third training image is the image for including the second feature, the 4th training image be include described
The image of fisrt feature；
Second training image is input to the second generation model for expression and obtained by second objective function
The 5th training image be input to obtained the 6th training image of the first generation model again, with second training image
Difference, the 5th training image is the image for including the fisrt feature, the 6th training image be include described second
The image of feature.
Optionally, the first object function
laid 0=| | x0-G1(G0(x0))||1
Wherein, x0Indicate first training image, G0Indicate that described first generates model, G0(x0) indicate the third
Training image, G1Indicate that described second generates model, G1(G0(x0)) indicate the 4th training image, | | x0-G1(G0(x0))|
|1It indicates to x0-G1(G0(x0)) calculate L1 norm.
Optionally, second objective function
laid 1=| | x1-G0(G1(x1))||1
Wherein, x1Indicate second training image, G1Indicate that described second generates model, G1(x1) indicate the described 5th
Training image, G0Indicate that described first generates model, G0(G1(x1)) indicate the 6th training image, | | x1-G0(G1(x1))|
|1It indicates to x1-G0(G1(x1)) calculate L1 norm.
Optionally, the fisrt feature is that target object has hot spot on the face, and the second feature is that target object does not have on the face
There is hot spot.
Optionally, the fisrt feature is that target object is worn glasses on the face, the second feature be target object on the face not
It wears glasses.
In the embodiment of the present application, have since the first generation model can become the image procossing with fisrt feature
The image of second feature, second, which generates model, can become the figure with fisrt feature for the image procossing with second feature
Picture, if the image restoring crossed by the first generation model treatment can be original image by the second generation model, first is generated
The image restoring crossed by the second generation model treatment can be original image by model, it may be considered that first generates model
It is more accurate with specific aim processing of the second generation model to image, it is based on this, can be arranged for the training of image processing model
It minimizes first object function and/or minimizes the second bound for objective function, utilize model optimization algorithm more new images
Model is handled, the image processing model enable is accurate to handle image, and the picture quality of output is also higher,
Improve user experience.
Based on a kind of image processing method that above embodiments provide, the embodiment of the present application also provides a kind of image procossings
Its working principle is described in detail with reference to the accompanying drawing in device.
Embodiment five
Referring to Fig. 8, which is a kind of structural block diagram for image processing apparatus that the embodiment of the present application five provides.The present embodiment
The image processing apparatus of offer includes:
First image acquisition unit 210, for obtaining the first image, the first image includes fisrt feature；
Second image acquisition unit 220 obtains the second figure for the first image to be input in the first generation model
Picture, second image includes second feature, and the fisrt feature and the second feature are for same attribute and to have not
With the feature presented, the first generation model uses a kind of model training method such as provided by the embodiments of the present application to be trained
It obtains.
As first generation model be according to the embodiment of the present application one provide model training method training made of, can
Targetedly the first image is handled, the accuracy of image procossing is higher, and the picture quality of output is also higher, therefore improves
User experience.
Embodiment six
Referring to Fig. 9, which is the structural block diagram for another image processing apparatus that the embodiment of the present application six provides.This implementation
Example provide image processing apparatus include:
Third image acquisition unit 310, for obtaining third image, the third image includes second feature；
4th image acquisition unit 320 obtains the 4th figure for the third image to be input in the second generation model
Picture, described image include fisrt feature, and the fisrt feature and the second feature, which are for same attribute and with difference, is in
Existing feature, the second generation model use a kind of model training method such as provided by the embodiments of the present application to be trained
It arrives.
As second generation model be according to the embodiment of the present application one provide model training method training made of, can
Targetedly third image is handled, the accuracy of image procossing is higher, and the picture quality of output is also higher, therefore improves
User experience.
The description and claims of this application and term " first ", " second ", " third ", " in above-mentioned attached drawing
The (if present)s such as four " are to be used to distinguish similar objects, without being used to describe a particular order or precedence order.It should manage
The data that solution uses in this way are interchangeable under appropriate circumstances, so that the embodiments described herein can be in addition to illustrating herein
Or the sequence other than the content of description is implemented.In addition, term " includes " and " having " and their any deformation, it is intended that
Cover it is non-exclusive include, for example, containing the process, method, system, product or equipment of a series of steps or units need not limit
In step or unit those of is clearly listed, but may include be not clearly listed or for these process, methods, produce
The other step or units of product or equipment inherently.
It is apparent to those skilled in the art that for convenience and simplicity of description, the system of foregoing description,
The specific work process of device and unit, can refer to corresponding processes in the foregoing method embodiment, and details are not described herein.
In several embodiments provided herein, it should be understood that disclosed system, device and method can be with
It realizes by another way.For example, the apparatus embodiments described above are merely exemplary, for example, the unit
It divides, only a kind of logical function partition, there may be another division manner in actual implementation, such as multiple units or components
It can be combined or can be integrated into another system, or some features can be ignored or not executed.Another point, it is shown or
The mutual coupling, direct-coupling or communication connection discussed can be through some interfaces, the indirect coupling of device or unit
It closes or communicates to connect, can be electrical property, mechanical or other forms.
The unit as illustrated by the separation member may or may not be physically separated, aobvious as unit
The component shown may or may not be physical unit, it can and it is in one place, or may be distributed over multiple
In network unit.It can select some or all of unit therein according to the actual needs to realize the mesh of this embodiment scheme
's.
It, can also be in addition, each functional unit in each embodiment of the application can integrate in one processing unit
It is that each unit physically exists alone, can also be integrated in one unit with two or more units.Above-mentioned integrated list
Member both can take the form of hardware realization, can also realize in the form of software functional units.
If the integrated unit is realized in the form of SFU software functional unit and sells or use as independent product
When, it can store in a computer readable storage medium.Based on this understanding, the technical solution of the application is substantially
The all or part of the part that contributes to existing technology or the technical solution can be in the form of software products in other words
It embodies, which is stored in a storage medium, including some instructions are used so that a computer
Equipment (can be personal computer, server or the network equipment etc.) executes the complete of each embodiment the method for the application
Portion or part steps.And storage medium above-mentioned includes: USB flash disk, mobile hard disk, read-only memory (ROM, Read-Only
Memory), random access memory (RAM, Random Access Memory), magnetic or disk etc. are various can store journey
The medium of sequence code.
Those skilled in the art are it will be appreciated that in said one or multiple examples, function described in the invention
It can be realized with hardware, software, firmware or their any combination.It when implemented in software, can be by these functions
Storage in computer-readable medium or as on computer-readable medium one or more instructions or code transmitted.
Computer-readable medium includes computer storage media and communication media, and wherein communication media includes convenient for from a place to another
Any medium of one place transmission computer program.Storage medium can be general or specialized computer can access it is any
Usable medium.
Above-described specific embodiment has carried out further the purpose of the present invention, technical scheme and beneficial effects
It is described in detail, it should be understood that the foregoing is merely a specific embodiment of the invention.
The above, above embodiments are only to illustrate the technical solution of the application, rather than its limitations；Although referring to before
Embodiment is stated the application is described in detail, those skilled in the art should understand that: it still can be to preceding
Technical solution documented by each embodiment is stated to modify or equivalent replacement of some of the technical features；And these
It modifies or replaces, the range of each embodiment technical solution of the application that it does not separate the essence of the corresponding technical solution.
Claims (10)
1. a kind of model training method, which is characterized in that the described method includes:
Training image is obtained, the training image includes the first training image and/or the second training image, wherein first instruction
Practicing image is the image for including fisrt feature, and second training image is the image for including second feature；
Image processing model is trained using the training image, wherein described image processing model includes the first generation mould
Type and second generates model；
Training process includes:
First training image input the first generation model is obtained into third training image, by the third training image
It inputs the second generation model and obtains the 4th training image, the third training image is the figure for including the second feature
Picture, the 4th training image is the image for including the fisrt feature；
The training process further includes following constraint condition: first object function minimization, and the first object function is used for table
Difference up between the 4th training image and first training image；
And/or training process includes:
Second training image input the second generation model is obtained into the 5th training image, by the 5th training image
It inputs the first generation model and obtains the 6th training image, the 5th training image is the figure for including the fisrt feature
Picture, the 6th training image is the image for including the second feature；
The training process further includes following constraint condition: the second the minimization of object function；
Second objective function is used to express the difference of the 6th training image and second training image.
2. the method according to claim 1, wherein the fisrt feature be target object have hot spot on the face, institute
Stating second feature is target object on the face without hot spot.
3. method according to claim 1 or 2, which is characterized in that the fisrt feature is that target object is worn glasses on the face,
The second feature is that target object is not worn glasses on the face.
4. a kind of model training apparatus, which is characterized in that described device includes:
Training image acquiring unit, for obtaining training image, the training image includes the first training image and/or the second instruction
Practice image, wherein first training image is the image for including fisrt feature, second training image be include the second spy
The image of sign, the fisrt feature and the second feature are for same attribute and with the feature of different presentations；
Model training unit, for being trained using the training image to image processing model, wherein described image is handled
Model includes that the first generation model and second generate model；
Training process includes:
First training image input the first generation model is obtained into third training image, by the third training image
It inputs the second generation model and obtains the 4th training image, the third training image is the figure for including the second feature
Picture, the 4th training image is the image for including the fisrt feature；
Second training image input the second generation model is obtained into the 5th training image, by the 5th training image
It inputs the first generation model and obtains the 6th training image, the 5th training image is the figure for including the fisrt feature
Picture, the 6th training image is the image for including the second feature；
The training process further includes following constraint condition: first object function minimization and/or the second the minimization of object function；
The first object function is used to express the difference between the 4th training image and first training image, described
Second objective function is used to express the difference of the 6th training image and second training image.
5. device according to claim 4, which is characterized in that the fisrt feature is that target object has hot spot on the face, institute
Stating second feature is target object on the face without hot spot.
6. device according to claim 4 or 5, which is characterized in that the fisrt feature is that target object is worn glasses on the face,
The second feature is that target object is not worn glasses on the face.
7. a kind of image processing method, which is characterized in that the described method includes:
The first image is obtained, the first image includes fisrt feature；
The first image is input in the first generation model, the second image is obtained, second image includes second feature,
The fisrt feature and the second feature are to generate model for same attribute and with the feature of different presentations, described first
It is trained to obtain using model training method as described in any one of claims 1-3.
8. a kind of image processing apparatus, which is characterized in that described device includes:
First image acquisition unit, for obtaining the first image, the first image includes fisrt feature；
Second image acquisition unit obtains the second image for the first image to be input in the first generation model, described
Second image includes second feature, and the fisrt feature and the second feature are for same attribute and with different presentations
Feature, the first generation model are trained to obtain using model training method as described in any one of claims 1-3.
9. a kind of image processing method, which is characterized in that the described method includes:
Third image is obtained, the third image includes second feature；
The third image is input in the second generation model, obtains the 4th image, described image includes fisrt feature, described
Fisrt feature and the second feature are for same attribute and with the feature of different presentations, and described second, which generates model, uses
Model training method as described in any one of claims 1-3 is trained to obtain.
10. a kind of image processing apparatus, which is characterized in that described device includes:
Third image acquisition unit, for obtaining third image, the third image includes second feature；
4th image acquisition unit obtains the 4th image for the third image to be input in the second generation model, described
Image includes fisrt feature, and the fisrt feature and the second feature are for same attribute and with the spy of different presentations
Sign, the second generation model are trained to obtain using model training method as described in any one of claims 1-3.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910280851.9A CN110009044B (en) | 2019-04-09 | 2019-04-09 | Model training method and device, and image processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910280851.9A CN110009044B (en) | 2019-04-09 | 2019-04-09 | Model training method and device, and image processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110009044A true CN110009044A (en) | 2019-07-12 |
| CN110009044B CN110009044B (en) | 2021-09-03 |
Family
ID=67170545
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910280851.9A Active CN110009044B (en) | 2019-04-09 | 2019-04-09 | Model training method and device, and image processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110009044B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107220929A (en) * | 2017-06-23 | 2017-09-29 | 深圳市唯特视科技有限公司 | A kind of non-paired image method for transformation using the consistent confrontation network of circulation |
| CN107577985A (en) * | 2017-07-18 | 2018-01-12 | 南京邮电大学 | The implementation method of the face head portrait cartooning of confrontation network is generated based on circulation |
| US20180336471A1 (en) * | 2017-05-19 | 2018-11-22 | Mehdi Rezagholizadeh | Semi-supervised regression with generative adversarial networks |
| CN109426858A (en) * | 2017-08-29 | 2019-03-05 | 京东方科技集团股份有限公司 | Neural network, training method, image processing method and image processing apparatus |
| WO2019056000A1 (en) * | 2017-09-18 | 2019-03-21 | Board Of Trustees Of Michigan State University | Disentangled representation learning generative adversarial network for pose-invariant face recognition |
-
2019
- 2019-04-09 CN CN201910280851.9A patent/CN110009044B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180336471A1 (en) * | 2017-05-19 | 2018-11-22 | Mehdi Rezagholizadeh | Semi-supervised regression with generative adversarial networks |
| CN107220929A (en) * | 2017-06-23 | 2017-09-29 | 深圳市唯特视科技有限公司 | A kind of non-paired image method for transformation using the consistent confrontation network of circulation |
| CN107577985A (en) * | 2017-07-18 | 2018-01-12 | 南京邮电大学 | The implementation method of the face head portrait cartooning of confrontation network is generated based on circulation |
| CN109426858A (en) * | 2017-08-29 | 2019-03-05 | 京东方科技集团股份有限公司 | Neural network, training method, image processing method and image processing apparatus |
| WO2019056000A1 (en) * | 2017-09-18 | 2019-03-21 | Board Of Trustees Of Michigan State University | Disentangled representation learning generative adversarial network for pose-invariant face recognition |
Non-Patent Citations (4)
| Title |
|---|
| JUN-YAN ZHU ET AL: "Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks", 《2017 IEEE INTERNATIONAL CONFERENCE ON COMPUTER VISION》 * |
| LVMIN ZHANG ET AL: "Style Transfer for Anime Sketches with Enhanced Residual U-net and Auxiliary Classifier GAN", 《2017 4TH IAPR ASIAN CONFERENCE ON PATTERN RECOGNITION (ACPR)》 * |
| 许哲豪等: "基于生成对抗网络的图片风格迁移", 《软件导刊》 * |
| 谢志峰等: "基于生成对抗网络的HDR图像风格迁移技术", 《上海大学学报》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110009044B (en) | 2021-09-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP7516260B2 (en) | A skeletal system for animating virtual avatars | |
| CN111914629B (en) | Method, device, equipment and storage medium for generating training data for face recognition | |
| US20180365484A1 (en) | Head-mounted display with facial expression detecting capability | |
| CN108229508B (en) | Training apparatus and training method for training image processing apparatus | |
| CN110111246B (en) | Virtual head portrait generation method and device and storage medium | |
| TWI770542B (en) | Face image processing method, image equipment and storage medium | |
| CN104978764A (en) | Three-dimensional face mesh model processing method and three-dimensional face mesh model processing equipment | |
| CN105637512A (en) | Method and system to create custom products | |
| CN106686365A (en) | Lens adjusting method and lens adjusting device for head-mounted display equipment, and head-mounted display equipment | |
| EP4073682B1 (en) | Generating videos, which include modified facial images | |
| CN108449596B (en) | 3D stereoscopic image quality evaluation method integrating aesthetics and comfort | |
| CN111936912A (en) | Head scan alignment using eye registration | |
| CN107302694A (en) | Method, equipment and the virtual reality device of scene are presented by virtual reality device | |
| CN113327191A (en) | Face image synthesis method and device | |
| CN113850169A (en) | Face attribute migration method based on image segmentation and generation of confrontation network | |
| CN114904268A (en) | Virtual image adjusting method and device, electronic equipment and storage medium | |
| CN110321964A (en) | Identification model update method and relevant apparatus | |
| CN104933742A (en) | Automatic cartoon image generation method | |
| CN110009044A (en) | A kind of model training method and device, image processing method and device | |
| CN111062899A (en) | Guidance-based blink video generation method for generating confrontation network | |
| CN109584145A (en) | Cartoonize method and apparatus, electronic equipment and computer storage medium | |
| CN105974582A (en) | Method and system for image correction of head-wearing display device | |
| CN112991152A (en) | Image processing method and device, electronic equipment and storage medium | |
| CN110444062A (en) | A kind of cloud VR learning system and method with eyesight protection function | |
| CN113866987A (en) | Method for interactively adjusting interpupillary distance and image surface of augmented reality helmet display by utilizing gestures |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |