CN112884673A - Reconstruction method for missing information between coffin chamber mural blocks of improved loss function SinGAN - Google Patents

Abstract

The invention discloses a reconstruction method for missing information among tomb mural blocks of an improved loss function SinGAN, which comprises the following steps: taking any one mural block image I in mural set_rThen down-sampled to { I_r0,I_r1,I_r2,I_r3,I_r4In which I_m4The murals are masked and sampled for 4 times; input I to the bottom layer of the generator_m4'Jing' generator G₄To produce an outwardly-extending mural G₄(I_m4) G is₄(I_m4) And I_r4Input to a discriminator D₄Comparing and judging, and updating the weight parameter of the layer according to the loss function; obtaining weight parameters of each layer in the generator; training the generator by using a training set and a discriminator to enable the generator and the discriminator to reach Nash equilibrium, and then reconstructing missing information among the coffin chamber mural blocks by using the trained generator.

Description

Reconstruction method for missing information between coffin chamber mural blocks of improved loss function SinGAN

Technical Field

The invention belongs to the field of digital image restoration, and relates to a reconstruction method for missing information among coffin chamber mural blocks of an improved loss function SinGAN.

Background

The coffin chamber wall painting has double important meanings to the ancient history and the artistic history of China, truly reflects the living condition, the social fashion and the artistic interest of the ancient Chinese, and simultaneously reflects the religious belief, the funeral culture and other conditions of people. As the coffin chamber wall painting is different from the palace chamber wall painting and the stone cave temple wall painting, the coffin chamber wall painting is buried underground for thousands of years, is completely a closed space before being excavated, and has strong reliability of residual information. And the breadth of the coffin chamber wall painting is larger, taking the chapter and radix pseudostellariae tomb ball picture as an example: the mural is 7 meters long and 3 meters high. Under the archaeological excavation condition in the seventies and eighties of the last century, the mode of taking off the archaeological excavation by blocks can be only adopted. Therefore, a large number of precious coffin chamber murals stored in blocks are generated, and the inter-block information loss of the murals is generated in the uncovering process, so that the continuity and the integrity of the whole murals are influenced.

In the conventional method for repairing digital information of computer-aided ancient murals, digital Image inpainting is used for finally completing reconstruction of mural information by retrieving the edge of an Image information missing area, diffusing Image residual information from the edge and filling the Image residual information into the missing area, and filling information holes layer by layer like onion peeling. The technology is mainly completed from two technical directions, on one hand, the technology is a PDE model based on pixel diffusion, the high-order partial derivative function of a filling front edge is calculated, and information filling is completed by using different diffusion equations, so that the defect is that a fuzzy phenomenon is generated when picture information is greatly lost; another aspect is a texture synthesis model based on sample filling that completes information completion in a certain filling order by comparing the correlation of the remaining samples with the filling front, with the disadvantage that filling of a large number of similar samples can create a mosaic effect. The two methods have the defects that when the deletion area is large, a good repairing effect of the coffin chamber mural cannot be achieved, available information is only searched from the existing residual information, the limitation is large, holes can only be filled inwards for the block murals, and the requirement for rebuilding the extensional information among the block murals cannot be met.

Disclosure of Invention

The present invention is directed to overcoming the above-mentioned shortcomings of the prior art, and providing a method for reconstructing missing information between mural blocks of a coffin chamber by improving a loss function SinGAN, which can achieve the reconstruction of the extensional information between mural blocks.

In order to achieve the above object, the method for reconstructing missing information between coffin chamber wall painting blocks of the improved loss function SinGAN of the present invention comprises the following steps:

1) acquiring images of a plurality of blocks of a mural excavated in a tomb road in blocks to obtain a mural block image, adding a mask to the mural block image, standardizing the mural block image for min-max, constructing an image set through the standardized mural block image, and using the image set as a SinGAN to generate a training set of ductility information between the murals;

2) constructing a generating network based on SinGAN and containing generators and discriminators with 5 scales, and increasing reconstruction loss L_recPixel reconstruction loss and texture loss L_textureA loss function of (d);

3) taking any one mural block image I in mural set_rThen down-sampled to { I_r0,I_r1,I_r2,I_r3,I_r4In which I_m4The murals are masked and sampled for 4 times;

4) input I to the bottom layer of the generator_m4'Jing' generator G₄To produce an outwardly-extending mural G₄(I_m4) G is₄(I_m4) And I_r4Input to a discriminator D₄Comparing and judging, and updating the weight parameter of the layer according to the loss function;

5) repeating the step 4) to obtain the weight parameters of each layer in the generator;

6) training the generator by using a training set and a discriminator to enable the generator and the discriminator to reach Nash equilibrium, and then reconstructing missing information among the coffin chamber mural blocks by using the trained generator.

Using pixel reconstruction loss to capture the whole pixel information around the mural generation area, i.e.

Where M is a binary mask, I_rnFor downsampling murals of real data, G_nTo generate a mural picture generated by the network,

multiplication is performed on the mural pixels;

texture loss is designed using Gram matrices, i.e.

Wherein G is a Gram matrix, l_nFor real mural images I_rnOutput through the nth layer of the multiscale generator, I_nFor a multi-scale generator G_nThe generated mural, g represents the generated network;

the reconstruction loss is designed as:

L_rec＝||G_n(Noize+I_mn)-I_rn||² (4)

wherein G is_nIs the nth layer of the multi-scale generator, (Noize + I)_mn) For superposition of the noise of each layer in the multiscale generator with the output mural of the previous layer, I_rnIs a real mural.

Reconstructed loss L_recPixel reconstruction loss and texture loss L_textureThe loss functions of (a) are:

L_rec＝||G_n(Noize+I_mn)-I_rn|| (6)

L_G＝χL_rec+L_2+texture (7)

wherein L is_2+textureIs the sum of the pixel reconstruction loss and texture loss, L, of the mural_DAs a loss function of the discriminator, L_DFor updating the discriminator weight, χ being a tunable parameter, L is adjusted by adjusting the loss function of the generator_2+texture。

Design the structure of multi-scale generator and discriminator, let { I_r0,I_r1,I_r2...I_rnRespectively obtaining a group of mural block images I obtained by down-sampling real murals_mnFor downsampling n times of mural block images after masking, I is input at the lowest layer of the multi-scale generator_mnGo through generator G_nGenerating mural G_n(I_mn) Then inputted to a discriminator D_nNeutralization of_rnComparing and judging, updating the weight parameter of the layer, and comparing G_n(I_mn) Upsampling and adding noise with the same size as the upsampled noise to form a generator G_n-1The input image is cycled to the original size of the image.

SSIM is used to measure fidelity and similarity between two murals, wherein,

wherein,

for mural painting original drawing I_rThe average value of (a) of (b),

for extended fresco I_mThe average value of (a) of (b),

is an original wall painting I_rThe variance of (a) is determined,

for extended rear fresco I_mThe variance of the measured values is calculated,

is I_r,I_mCovariance of (a), b₁And b₂Used for preventing the overflow caused by the fact that the denominator of the SSIM is zero, the SSIM is positioned at 0,1]Wherein, when the SSIM is larger, the content of the two images is more similar.

The invention has the following beneficial effects:

the invention relates to a reconstruction method for missing information among graveyard mural blocks of an improved loss function SinGAN, which is characterized in that during specific operation, the SinGAN after the loss function is improved is adopted to generate extensional information at two sides of the mural blocks, the reconstruction of the missing part among the mural blocks excavated out of earth by an underground graveyard is realized, and compared with the traditional digital image restoration method for filling the information of the graveyard murals by adopting an image inpainting method, the reconstruction method is more suitable for generating the picture information by the outward extensibility of the mural blocks.

Drawings

FIG. 1 is a flow chart of the present invention for generating a discriminant;

FIG. 2 is a diagram of a trained mural according to the present invention;

FIG. 3 is a wall drawing of the present invention;

FIG. 4 is a diagram of the result of SSIM value calculation in the present invention;

FIG. 5 is a diagram showing the results of mural generation according to the present invention.

Detailed Description

The invention is described in further detail below with reference to the accompanying drawings:

referring to fig. 1, the method for reconstructing missing information between coffin chamber mural blocks of the improved loss function SinGAN according to the present invention comprises the following steps:

1) the method comprises the steps of collecting images of the high-definition coffin chamber murals, adding masks to the collected images, then conducting min-max standardization, then constructing 8000 images with 256 x 256 through the standardized murals, using the mural sets as training sets, collecting the high-definition coffin chamber murals through shot segmentation, and meanwhile selecting the standardized murals to guarantee redundancy removal maximization performance of effective information in the shot segmentation.

The calibration method comprises the following steps of firstly, obtaining a calibration set of a wall painting of a coffin chamber, wherein the calibration set comprises a calibration set of calibration data, a calibration set of calibration data and a calibration set of calibration data. The preprocessing process is to give a training fresco I_rStandardizing the mural to I_r∈[0,1]^256*256*3The formula for min-max normalization is:

wherein, x is the numerical value of the sample point, Min is the minimum value of the sample data, and Max is the maximum value of the sample data.

And subtracting the minimum value from the value of the sample point, and dividing the value by the difference value between the maximum value and the minimum value of the sample point so as to improve the performance of the neural network.

While performing the training, I_rFor the original mural, a mask M ∈ [0,1] is defined]^256*256*3So that M_ijThe central part of the image can be masked, wherein the mask is a binary mask M (mask) using only two values of 0 and 1, the value of 1 indicates that the corresponding part in the image is reserved, and the value of 0 indicates that the corresponding part in the image needs to be completed.

2) Constructing 5 multi-scale generators and discriminators of the SinGAN, and adding a loss function for optimizing the iteration degree;

designing 5 multi-scale generators and discriminators, wherein each generator and discriminator is provided with 6 convolution layers, the convolution kernel is 4 x 4, two layers of expansion convolutions are used for completing mural images, each discriminator is composed of 6 layers of full convolution, the generator at the bottom can generate the integrity information of the mural, and the more the generator generates details each layer up to the generator G at the last layer₀The size of the generation is 256 × 256 and the content and style of the mural will become more and more realistic.

The method comprises the following steps of capturing whole pixel information around a mural generation area by adopting pixel reconstruction loss, wherein the expression is as follows:

where M is a binary mask, I_rnSome down-sampled murals for real data, G_nTo generate a mural picture generated by the network,

is a mural pixel multiplication.

Texture loss is designed by the Gram matrix, namely:

wherein G is a Gram matrix, l_nFor real mural images I_rnOutput through the nth layer of the network, I_nFor a multi-scale generator G_nAnd g represents a generated network, and the difference of the texture characteristics of each layer is extracted, so that the epitaxial mural image has the same local texture as the global texture, thereby enhancing the reality of the epitaxial mural.

Design reconstruction loss, i.e.

L_rec＝||G_n(Noize+I_mn)-I_rn||² (4)

Wherein G is_nIs the nth layer of the multi-scale generator, (Noize + I)_mn) For each layer of noise superimposed with the output fresco of the previous layer, I_rnIs a real mural.

3) Taking a wall painting I in the wall painting collection_rDown-sampled to { I_r0,I_r1,I_r2,I_r3,I_r4}，I_m4For the mural with 4 times of down-sampling after masking, I is input at the lowest layer of the multi-scale generator_m4Go through generator G₄To form a mural G₄(I_m4)；

4) G is to be₄(I_m4) And I_r4Input to a discriminator D₄Comparing and judging, updating the weight parameter of the layer according to the trained loss function, and repeating until G₀Layers, since each layer has an update of the weight parameter, the resulting wallThe painting will become more and more realistic.

The generator can extract the mural characteristics to generate the mural, and the training set mural I_rDown-sampling 4 times to obtain { I_r0,I_r1,I_r2,I_r3,I_r4I after mask is defined_mDown-sampling 4 times to obtain I_m4. First, to the generator G₄Inputting a mural I_m4The generator is able to generate the overall information of the mural. Will generate a result G₄(I_m4) And I_r4Input to a discriminator D₄Judging, updating the weight parameter of the layer according to the loss function, and obtaining G₄(I_m4) Up-sampling to obtain I_m3And superimposed with noise of the same size, the superimposed fresco being input to G₃Generator to get G₃(I_m3) And I_r3Input discriminator D₃Performing discrimination, calculating loss updating parameters, and repeating until G₀Until the generator, the details and contents of the generated mural effect are richest, the quality of the generated mural is greatly improved, and the generation and judgment flow chart is shown in fig. 1:

5) through 200 epoch cycles, the generator and the discriminator reach Nash equilibrium to complete the training of the model, and the trained model is used for reconstructing the missing information among the tomb room mural blocks.

The generator and the discriminator are both trained by a loss function, which is:

L_rec＝||G_n(Noize+I_mn)-I_rn|| (6)

L_G＝χL_rec+L_2+texture (7)

L_2+texturefor the sum of mural pixel reconstruction loss and texture loss, for updating generator weights to train the generator, L_DAs a loss function of the discriminator, and then according to L_DUpdating the weight of the discriminator, finally carrying out countermeasure training on the discriminator and the generator, setting x as an adjustable parameter to be 0.96, and adjusting L_GLoss function of generator to adjust L_2+textureAnd fixing the weight parameters after the training of each layer is finished, and then training the next layer.

After training is finished, an SSIM method is used for evaluating a generated picture, the numerical value is between [0 and 1], the larger the numerical value is, the higher the similarity with an original picture is, the SSIM is used for measuring the fidelity and the similarity between two murals, the SSIM is not only used for sensing errors, but also can be used for observing the distortion degree of a part relevant to sensing in a visual system, the similarity between two murals with different sizes can be evaluated, and the formula is as follows:

wherein,

for mural painting original drawing I_rThe average value of (a) of (b),

for extended fresco I_mThe average value of (a) of (b),

is an original wall painting I_rThe variance of the measured values is calculated,

for extended rear fresco I_mThe variance of the measured values is calculated,

is I_r,I_mCovariance of (a), b₁And b₂For preventing overflow due to SSIM denominator being zeroIn this case, the SSIM score is [0,1]]In between, when the contents of the two images are completely the same, the SSIM score is equal to 1.

Comparing a trained mural with the original mural to obtain the SSIM value shown in FIG. 4, wherein the SSIM result shows that the model is rich in the content and texture of the generated mural, the trained weight is stored, and a mural picture is input to obtain the generated result shown in FIG. 5.

Claims (5)

1. A reconstruction method for missing information among coffin chamber mural blocks of an improved loss function SinGAN is characterized by comprising the following steps:

1) acquiring images of a plurality of blocks of a mural excavated in a tomb road in blocks to obtain a mural block image, adding a mask to the mural block image, standardizing the mural block image for min-max, constructing an image set through the standardized mural block image, and using the image set as a SinGAN to generate a training set of ductility information between the murals;

2) constructing a generating network based on SinGAN and containing generators and discriminators with 5 scales, and increasing reconstruction loss L_recPixel reconstruction loss and texture loss L_textureA loss function of (d);

3) taking any one mural block image I in mural set_rThen down-sampled to { I_r0,I_r1,I_r2,I_r3,I_r4In which I_m4The murals are masked and sampled for 4 times;

4) input I to the bottom layer of the generator_m4'Jing' generator G₄To produce an outwardly-extending mural G₄(I_m4) G is₄(I_m4) And I_r4Input to a discriminator D₄Comparing and judging, and updating the weight parameter of the layer according to the loss function;

5) repeating the step 4) to obtain the weight parameters of each layer in the generator;

6) training the generator by using a training set and a discriminator to enable the generator and the discriminator to reach Nash equilibrium, and then reconstructing missing information among the coffin chamber mural blocks by using the trained generator.

2. A method as claimed in claim 1, wherein the loss of information in the neighborhood of the mural generation area is determined by applying pixel reconstruction loss to capture the global pixel information surrounding the mural generation area

Where M is a binary mask, I_rnFor downsampling murals of real data, G_nTo generate a mural picture generated by the network,

multiplication is performed on the mural pixels;

texture loss is designed using Gram matrices, i.e.

Wherein G is a Gram matrix, l_nFor real mural images I_rnOutput through the nth layer of the multiscale generator, I_nFor a multi-scale generator G_nThe generated mural, g represents the generated network;

the reconstruction loss is designed as:

L_rec＝||G_n(Noize+I_mn)-I_rn||² (4)

wherein G is_nIs the nth layer of the multi-scale generator, (Noize + I)_mn) For superposition of the noise of each layer in the multiscale generator with the output mural of the previous layer, I_rnIs a real mural.

3. Method for reconstructing missing information between coffin chamber wall blocks of the modified loss function SinGAN as claimed in claim 1, wherein the reconstructed loss L is_recPixel reconstruction loss and textureLoss L_textureThe loss functions of (a) are:

L_rec＝||G_n(Noize+I_mn)-I_rn|| (6)

L_G＝χL_rec+L_2+texture (7)

wherein L is_2+textureIs the sum of the pixel reconstruction loss and texture loss, L, of the mural_DAs a loss function of the discriminator, L_DFor updating the discriminator weight, χ being a tunable parameter, L is adjusted by adjusting the loss function of the generator_2+texture。

4. The method as claimed in claim 1, wherein the structure of the multi-scale generator and the discriminator is designed by { I }_r0,I_r1,I_r2...I_rnRespectively obtaining a group of mural block images I obtained by down-sampling real murals_mnFor downsampling n times of mural block images after masking, I is input at the lowest layer of the multi-scale generator_mnGo through generator G_nGenerating mural G_n(I_mn) Then inputted to a discriminator D_nNeutralization of_rnComparing and judging, updating the weight parameter of the layer, and comparing G_n(I_mn) Upsampling and adding noise with the same size as the upsampled noise to form a generator G_n-1The input image is cycled to the original size of the image.

5. The method for reconstructing information missing from blocks of a coffin chamber wall painting using the modified loss function SinGAN as claimed in claim 1, wherein SSIM is used to measure fidelity and similarity between two wall paintings,

wherein,

for mural painting original drawing I_rThe average value of (a) of (b),

for extended fresco I_mThe average value of (a) of (b),

is an original wall painting I_rThe variance of (a) is determined,

for extended rear fresco I_mThe variance of the measured values is calculated,

is I_r,I_mCovariance of (a), b₁And b₂Used for preventing the overflow caused by the fact that the denominator of the SSIM is zero, the SSIM is positioned at 0,1]Wherein, when the SSIM is larger, the content of the two images is more similar.

Images