CN111696028A - Method and device for processing cartoon of real scene image, computer equipment and storage medium - Google Patents

Abstract

The present application relates to a method, device, computer equipment and storage medium for processing a real scene image cartoon based on artificial intelligence. The method includes: acquiring a real scene image; performing image reconstruction processing and abstract smoothing processing on the real scene image based on the semantic information of the real scene image to obtain an abstract cartoon image in which the real scene image is mapped on the cartoon domain; the abstract cartoon image has The salient structure in the real scene image, and the contour edge lines in the real scene image are missing; the abstract cartoon image is stylized to generate a style cartoon image with artistic style; the contour edge line of the style cartoon image is generated to obtain the real scene The cartoon image after the image is cartoonized. By adopting the method, the quality of the generated cartoon image can be improved.

Description

Processing method, device, computer equipment and storage for real scene image cartoonization medium

technical field

The present application relates to the technical field of artificial intelligence and the technical field of image processing, and in particular, to a processing method, device, computer equipment and storage medium for cartoonizing real scene images.

Background technique

With the rapid development of science and technology, various advanced image processing technologies continue to emerge, and the application scenarios of image processing technologies are becoming more and more extensive. For example, cartoonizing a real scene image belongs to one of the scenes.

The traditional cartoonization method is based on manually customized operators or detection parameters to cartoonize the real picture, without considering the information of the picture itself, it only performs image transformation in a fixed manner, and does not involve related artistic processing. The resulting cartoon images are of poor quality. Then, how to use other technologies (for example, artificial intelligence technology) to generate higher-quality cartoon images is a question worth thinking about.

SUMMARY OF THE INVENTION

Based on this, it is necessary to provide a processing method, apparatus, computer equipment and storage medium for cartoonizing a real scene image that can improve the image quality in view of the above technical problems.

A method for processing real scene image cartoonization, the method includes:

Get real scene images;

Based on the semantic information of the real scene image, image reconstruction and abstract smoothing are performed on the real scene image, and an abstract cartoon image is obtained by mapping the real scene image on the cartoon domain; the abstract cartoon image has the saliency structure, And the contour edge lines in the real scene image are missing;

Stylize abstract cartoon images to generate artistic style cartoon images;

The contour edge lines of the style cartoon image are generated, and the cartoon image of the real scene image after cartoonization is obtained.

A processing device for cartoonizing real scene images, the device comprises:

The acquisition module is used to acquire real scene images;

The abstract processing module is used to perform image reconstruction processing and abstract smoothing processing on the real scene image based on the semantic information of the real scene image, and obtain an abstract cartoon image in which the real scene image is mapped on the cartoon domain; the abstract cartoon image has the real scene image. The saliency structure in the real scene image is missing, and the contour edge lines in the real scene image are missing; the abstract cartoon image is stylized to generate a style cartoon image with artistic style;

The line generation module is used to generate the outline and edge lines of the style cartoon image, and obtain the cartoon image after cartoonization of the real scene image.

In one embodiment, the real scene images are sequentially extracted from a video frame sequence; the video frame sequence includes at least two real scene images arranged in sequence;

The device also includes:

The output module is used to generate a cartoon video file according to the cartoon image of each real scene image in the video frame sequence when the video frame sequence is the image frame sequence in the video file, or, when the video file is played , obtain and output the cartoon image corresponding to each real scene image in the video frame sequence in real time.

In one embodiment, the output module is further configured to, when the video frame sequence is an image frame sequence in the real-time video stream, output the cartoonized video stream in real time according to the cartoon image of the real scene image in the video frame sequence after cartoonization.

In one embodiment, the abstract processing module is further configured to input the real scene image into the trained deep neural network, so that in the first stage, semantic extraction is performed on the real scene image, and based on the extracted semantic information, the real scene image is extracted. Perform image reconstruction processing, and perform bilateral filtering and smoothing processing on the reconstructed image content to generate abstract cartoon images.

In one embodiment, the real scene image is input into the first generator of the abstract network in the deep neural network; the apparatus further comprises:

The model training module is used to obtain the sample data set; the sample data set includes the first set of real scene sample graphs; the sample data set is input into the deep neural network to be trained for iterative training, and the structural reconstruction loss value is determined in each iteration and bilateral filter smoothing loss value, and reconstruct the loss value and bilateral filtering smoothing loss value according to the structure to determine the target loss value; in each iteration, adjust the network model parameters according to the target loss value until the iteration is stopped, and the training is completed The network model parameters include the parameters of the first generator; the trained deep neural network includes the trained abstract network; wherein, the structural reconstruction loss value is used to represent the first generator according to the real scene sample map The difference in structural features between the generated cartoon image and the real scene sample image; the bilateral filtering smoothes the loss value, which is used to determine the difference between adjacent pixels in the cartoon image generated by the first generator according to the similarity of pixel values and the similarity of spatial positions. difference between.

In one embodiment, the abstract processing module is further configured to, in the first stage, perform stylization processing on the abstract cartoon image by the first generator to generate a style cartoon image with artistic style.

In one embodiment, the abstract network further includes a first discriminator; the network model parameters further include parameters of the first discriminator; the sample data set further includes a second set of abstract cartoon sample images; the model training module is further configured to perform at each round In the iteration, the structural reconstruction loss value, the bilateral filtering smoothing loss value and the style enhancement loss value are determined, and the target loss value is determined according to the structural reconstruction loss value, the bilateral filtering smoothing loss value and the style enhancement loss value; among them, the style enhancement loss The value is determined by the first discriminator according to the first probability output by the cartoon image generated by the first generator; the first probability is used to represent the probability that the cartoon image generated by the first generator belongs to the style of the abstract cartoon sample image.

In one embodiment, the model training module is further used to obtain a third set of original cartoon images; the original cartoon images in the third set have the same artistic style; according to the pre-trained line tracing network, the contours of each original cartoon image are extracted Edge lines; according to the extracted contour edge lines and the original cartoon image, an abstract cartoon sample image is generated, and the second set is obtained; the abstract cartoon sample image is an abstract image after the contour edge lines are eliminated from the original cartoon image.

In one embodiment, the deep learning network is a two-stage neural network, and the abstract network is a first-stage abstract network in the two-stage neural network; the two-stage neural network further includes a second-stage line drawing network; the line generation module is also used for In the second stage, the style cartoon image generated in the first stage is input to the second generator in the second stage line drawing network in the trained two-stage neural network, so as to pass the second generator to the style cartoon image The contour edge line generation process is performed to obtain a cartoon image after the real scene image is cartoonized; wherein, the second stage line drawing network is a deep neural network that generates contour edge lines in the second stage.

In one embodiment, the abstract cartoon sample image is an abstract image that eliminates contour edge lines from the original cartoon image; the sample data set further includes a third set of original cartoon images corresponding to the abstract cartoon sample image; the network model parameters It also includes the parameters of the second generator; the model training module is also used to determine the loss value of structural reconstruction, bilateral filter smoothing loss value, style enhancement loss value and edge line distribution loss value in each round of iteration, and according to the Structural reconstruction loss value, bilateral filter smoothing loss value, style enhancement loss value and edge line distribution loss value determine the target loss value; edge line distribution loss value is used to represent the value of the abstract cartoon sample image as the input of the second generator. The difference between the resulting image with lines and the original cartoon image corresponding to the abstract cartoon sample image.

In one embodiment, the second-stage line drawing network further includes a second discriminator; the network model parameters further include parameters of the second discriminator; the model training module is further configured to determine the structural reconstruction loss value, bilateral filter smoothing loss value, style enhancement loss value, edge line assignment loss value and edge line enhancement loss value; reconstruct loss value, bilateral filtering smoothing loss value, style enhancement loss value, edge line assignment loss value and said edge The line enhancement loss value determines the target loss value; the edge line enhancement loss value is the image with lines generated when the second generator takes the image generated by the first generator as input, and after inputting it to the second discriminator, The second probability is determined by the second probability output by the second discriminator; the second probability is used to characterize the line intensity difference between the image with lines generated by the second generator and the original cartoon image.

In one embodiment, the model training module is further configured to obtain a first set of real scene sample images, a second set of abstract cartoon sample images, and a third set of original cartoon images corresponding to the abstract cartoon sample images; The first set is input to the abstract network to be trained for structural reconstruction iterative training, the structural reconstruction loss value is determined in each round of structural reconstruction iterative training, and the parameters of the first generator are adjusted according to the structural reconstruction loss value until the initialization training stops. condition, to obtain the initialized abstract network; input the first set and the second set to the initialized abstract network, and input the second set and the third set to the second-stage line drawing network to be trained for network cascade iterative training, and at each Determine the structure reconstruction loss value, the bilateral filter smoothing loss value, the style enhancement loss value, and the edge line assignment loss value in the iterations; according to the structure reconstruction loss value, the bilateral filter smoothing loss value, the style enhancement loss value, and the edge line assignment Loss value, determine the comprehensive loss value, and according to the comprehensive loss value, adjust the parameters of the first generator and the first discriminator to initialize the abstract network, and adjust the second generator and second generator of the line drawing network in the second stage to be trained. The parameters of the discriminator until the iteration stops, and the trained two-stage neural network is obtained; the two-stage neural network includes the first-stage abstract network and the second-stage line drawing network.

A computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

Get real scene images;

Based on the semantic information of the real scene image, image reconstruction and abstract smoothing are performed on the real scene image, and an abstract cartoon image is obtained by mapping the real scene image on the cartoon domain; the abstract cartoon image has the saliency structure, And the contour edge lines in the real scene image are missing;

Stylize abstract cartoon images to generate artistic style cartoon images;

The contour edge lines of the style cartoon image are generated, and the cartoon image of the real scene image after cartoonization is obtained.

A computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

Get real scene images;

Based on the semantic information of the real scene image, image reconstruction and abstract smoothing are performed on the real scene image, and an abstract cartoon image is obtained by mapping the real scene image on the cartoon domain; the abstract cartoon image has the saliency structure, And the contour edge lines in the real scene image are missing;

Stylize abstract cartoon images to generate artistic style cartoon images;

The contour edge lines of the style cartoon image are generated, and the cartoon image of the real scene image after cartoonization is obtained.

The above-mentioned processing method, device, computer equipment and storage medium for cartoonizing real scene images deeply understand the key semantic information in real scene images, and based on the semantic information of real scene images, image reconstruction processing and abstraction are performed on real scene images. Smoothing, obtaining an abstract cartoon image of the real scene image mapped on the cartoon domain, combining image reconstruction and abstract smoothing, not only maintaining the salient structure in the real scene image, but also smoothing out insignificant details, making Colors and image content are smooth. Then, the abstract cartoon image is stylized to generate a style cartoon image with artistic style. It is equivalent to abstracting and smoothing the image in the first stage, as well as abstracting and stylizing the image content. Therefore, the contour edge lines are generated based on the style cartoon image obtained in the first stage, which can avoid the occurrence of inconsistencies in the non-salient region. Necessary lines, so that the lines can be clearly concentrated on the contour edge of the salient structure region, and then a cartoon image with higher image quality can be obtained.

Description of drawings

Fig. 1 is the application environment diagram of the processing method of real scene image cartoonization in one embodiment;

2 is a schematic flowchart of a processing method for cartoonizing a real scene image in one embodiment;

3 is a schematic diagram of the network structure of a two-stage neural network in one embodiment;

4 is a schematic diagram of a cartoon image in one embodiment;

5 is a schematic diagram of results in different states of an embodiment;

Fig. 6 is the comparison experiment effect diagram of this case and the reference group in one embodiment;

7 is a comparison diagram of cartoon images of different styles in one embodiment;

8 is a structural block diagram of a processing device for cartoonizing a real scene image in one embodiment;

9 is a structural block diagram of a processing device for cartoonizing a real scene image in another embodiment;

Figure 10 is an internal structure diagram of a computer device in one embodiment;

FIG. 11 is an internal structure diagram of a computer device in another embodiment.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present application more clearly understood, the present application will be described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application.

The processing method for cartoonizing a real scene image provided by the present application can be applied to the application environment shown in FIG. 1 . The terminal 102 communicates with the server 104 through the network. The terminal 102 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers and portable wearable devices, and the server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

The terminal 102 can obtain the real scene image input by the user or collected by itself, and send the real scene image to the server 104, and the server 104 executes the processing method for cartoonizing the real scene image in each embodiment of the present application. The server 104 can output the generated cartoon image to the terminal 102 for display.

It can be understood that the terminal 102 itself may also have the processing method for performing the cartoonization of real scene images in various embodiments of the present application, then the terminal 102 may directly convert the acquired real scene images into cartoon images, and output them for display. The execution subject that executes the methods in the embodiments of the present application is not limited here, that is, both the terminal 102 and the server 104 may execute.

It can be understood that the processing methods for cartoonizing real scene images in the embodiments of the present application are equivalent to using artificial intelligence technology to automatically convert real scene images into cartoon images.

Artificial intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technique of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making.

Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, including both hardware-level technology and software-level technology. The basic technologies of artificial intelligence generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation/interaction systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning/deep learning.

It can be understood that the processing methods for cartoonizing real scene images in various embodiments of the present application use computer vision technology in artificial intelligence technology to perform image reconstruction, smoothing and other abstract smoothing processing, stylization processing and stylization processing on real scene images. Image processing such as edge lines is generated, so that the real scene image is automatically converted into a cartoon image, and the cartoon processing of the real scene is realized.

Computer Vision Technology (Computer Vision, CV) Computer vision is a science that studies how to make machines "see". Further, it refers to the use of cameras and computers instead of human eyes to identify, track and measure targets. Machine vision, And further do graphics processing, so that computer processing becomes more suitable for human eye observation or transmission to the instrument detection image. As a scientific discipline, computer vision studies related theories and technologies, trying to build artificial intelligence systems that can obtain information from images or multidimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content/behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping It also includes common biometric identification technologies such as face recognition and fingerprint recognition.

In addition, the processing methods for cartoonizing real scene images in the embodiments of the present application also use the machine learning processing technology in the artificial intelligence technology. It can be understood that the embodiments of the present application are equivalent to using machine learning technology to understand image semantics, and to perform image reconstruction and abstract smoothing processing. In addition, some of the embodiments of the present application involve the training and use process of the deep learning network, and it is fully explained that the present application realizes the processing of cartoonizing real scene images based on machine learning.

Machine Learning (ML) is a multi-field interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in how computers simulate or realize human learning behaviors to acquire new knowledge or skills, and to reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent, and its applications are in all fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, teaching learning and other technologies.

In one embodiment, as shown in FIG. 2 , a method for processing real scene image cartoonization is provided, and the method is applied to a computer device as an example for illustration. The computer device may be the server or terminal in FIG. 1 , including The following steps:

Step 202, acquiring a real scene image.

The real scene image refers to an image in a real scene.

In one embodiment, the image content of the real scene image may include at least one of environment image content and object image content in the real scene.

It can be understood that the environmental image content is used to represent the image content of the background environment where the object is located. The object may be at least one of a person, an animal, an object, and the like. Therefore, the object image content may be at least one of person image content, animal image content, and object image content.

In one embodiment, the real scene image may be an individual picture, or may be an image in a sequence of video frames.

In one embodiment, the video frame sequence may be an image frame sequence in a video file that has been generated, that is, a real scene image, and may be a frame-by-frame image in the video file. Thereby realizing video cartoonization.

In one embodiment, the video frame sequence may also be an image frame sequence in a real-time video stream. That is, the real scene image may be a frame-by-frame image in a real-time video stream. Thereby realizing real-time video cartoonization.

The method in each embodiment of the present application is a method for cartoonizing a real scene image to convert it into a cartoon image. Cartoon image refers to an unreal image that is extracted from natural and real prototypes (ie, real scene images) and reproduced by artistic methods.

Step 204, based on the semantic information of the real scene image, perform image reconstruction processing and abstract smoothing processing on the real scene image, to obtain an abstract cartoon image in which the real scene image is mapped on the cartoon domain; the abstract cartoon image has significant features in the real scene image. Sexual structure, and lack of contour edge lines in real scene images.

Among them, the semantic information is the feature information of the image.

Image reconstruction processing refers to the process of reconstructing the generated image.

Abstract smoothing processing, that is, filtering processing, is used to eliminate irrelevant details in the image and abstract the image content into a cartoon image. It can be understood that since real scene images include rich image details, these image details should not be displayed in cartoon images, so abstract smoothing processing is performed to eliminate unnecessary interference and achieve smooth distribution of colors.

Saliency structure is the structural feature of key regions in real scene images. That is, the key image content in the real scene. The contour edge line refers to the edge contour line of the image content in the real scene image. It can be understood that each area in the real scene image has image content, and each part of the image content will have corresponding edge contour lines.

An abstract cartoon image refers to a non-real cartoon image that maps a real scene image in the real domain to a cartoon image in the cartoon domain. It can be understood that in the embodiment of the present application, performing image reconstruction processing on the real scene image based on the semantic information of the real scene image is equivalent to performing image reconstruction considering the key features in the real scene image, so that the generated abstraction can be reduced. The cartoon image has the saliency structure in the real scene image, and the combination of image reconstruction and abstract smoothing removes unnecessary details. Therefore, the abstract cartoon image lacks the unnecessary contour lines and lines in the real scene image. Details, become smoother.

Specifically, the computer device can input the real scene image into a pre-trained deep neural network to perform semantic extraction processing on the real scene image, and based on the extracted semantic information, perform image reconstruction processing on the real scene image, and then reconstruct the real scene image. After the image is abstracted and smoothed, the abstract cartoon image of the real scene image mapped on the cartoon domain is obtained.

In one embodiment, the above-mentioned pre-trained deep neural network may include at least two-stage neural networks. Therefore, the computer equipment can input the real scene image into the neural network of the first stage in the deep neural network. It will be appreciated that in other embodiments, the deep neural network may only be a single-stage neural network.

Step 206, performing stylization processing on the abstract cartoon image to generate a style cartoon image with artistic style.

Among them, the stylized processing refers to the processing of imparting artistic style to abstract cartoon images.

Artistic style refers to a comprehensive overall feature manifested in literary and artistic creations. It is the representative appearance of the work of art as a whole. Compared with the abstract cartoon image in step 204, an artistic style is added to the style cartoon image.

It can be understood that different artists have different artistic styles, so by stylizing the abstract cartoon images, a cartoon image with a style similar to the creative style of a certain artist can be generated. For example, an abstract cartoon image can be stylized into a cartoon image of a style similar to that created by Master Hayao Miyazaki.

It should be noted that the conversion of an abstract cartoon image into a style cartoon image with a fixed artistic style is not limited here. In fact, the abstract cartoon images can be stylized into stylized cartoon images with different artistic styles, respectively, as required.

In one embodiment, the computer device may also use the deep neural network in step 204 to stylize the abstract cartoon image to generate a stylized cartoon image with artistic style.

In another embodiment, the computer device may also use a preset stylized template to perform image overlay and fusion processing on the abstract cartoon image and the stylized template to generate a style cartoon image with artistic style. For example, an artistic style can be added to an abstract cartoon image by adding a filter (ie, a stylized template).

It can be understood that compared to abstract cartoon images, style cartoon images increase the information of artistic style, but style cartoon images still lack outline and edge lines.

Step 208 , generating the outline and edge lines of the style cartoon image, and obtaining the cartoon image after the real scene image is cartoonized.

Among them, the cartoon image after the real scene image is cartoonized is a cartoon image with outline edge lines, a salient structure in the real scene image, and an artistic style.

Specifically, the computer device may perform edge line generation processing on the style cartoon image to generate outline edge lines of the style cartoon image, so as to obtain a cartoon image after the real scene image is cartoonized.

In one embodiment, when the deep neural network in step 204 includes at least two stages of neural networks, the computer device can input the real scene image into the neural network of the second stage in the deep neural network, so as to adjust the style The cartoon image is processed by edge line generation.

In other embodiments, the computer device may also separately train a machine learning model dedicated to generating edge lines (ie, independent of the machine learning model in step 204 ) to generate outline edge lines of a cartoon-style image. In addition, the computer equipment can also use the edge detection operator to find the contour edge of the image in the style cartoon image, and generate the contour edge line, so as to obtain the cartoon image of the real scene image after cartoonization.

The above-mentioned processing method of real scene image cartoonization deeply understands the key semantic information in the real scene image, and based on the semantic information of the real scene image, image reconstruction processing and abstract smoothing processing are performed on the real scene image, and the real scene image map is obtained. For abstract cartoon images in the cartoon domain, image reconstruction and abstract smoothing are combined, which not only maintains the saliency structure in real scene images, but also smoothes out insignificant details, making the color and image content smooth. Then, the abstract cartoon image is stylized to generate a style cartoon image with artistic style. It is equivalent to abstracting and smoothing the image in the first stage (to smooth the non-salient structure), as well as the abstraction and stylization of the image content. Therefore, based on the style cartoon image obtained in the first stage, the contour edge lines are generated, Unnecessary lines can be avoided in non-salient regions, so that lines can be clearly concentrated on the contour edges of salient structural regions, and cartoon images with higher image quality can be obtained.

In one embodiment, the real scene images are sequentially extracted from a video frame sequence; the video frame sequence includes at least two real scene images arranged in sequence.

It can be understood that the video frame sequence may be an image frame sequence in a video file, or may be an image frame sequence in a real-time video stream.

Then, according to the methods in the embodiments of the present application, a corresponding cartoon image can be generated for each real scene image in the video frame sequence.

In one embodiment, the method further includes: when the video frame sequence is an image frame sequence in a video file, generating a cartoonized video file according to the cartoonized cartoon image of each real scene image in the video frame sequence .

Specifically, when the video frame sequence is an image frame sequence in a video file, the computer device may execute the method in each embodiment of the present application on each real scene image in the video frame sequence in sequence, and convert it into a cartoon image image. Then, a cartoonized video file is generated according to the cartoonized cartoon image of each real scene image.

It can be understood that this method is equivalent to generating a corresponding cartoonized video file by using the method in the embodiment of the present application before playing the video file, and the cartoonized video file can be directly played subsequently.

In one embodiment, the method further includes: when the video frame sequence is an image frame sequence in a video file, then when the video file is played, acquiring real-time images corresponding to each real scene in the video frame sequence corresponding to the cartoon image and output.

Specifically, in this embodiment, after converting each real scene image in the video frame sequence into a corresponding cartoon image, the computer device may not generate a separate cartoonized video file, but record the real scene Correspondence between images and cartoon images. Then, when the video file is subsequently played, that is, the video frame sequence corresponding to the video file is played, the computer device can output in real time the cartoon image cartoonized with each real scene image in the video frame sequence according to the corresponding relationship.

In one embodiment, when the computer device is a terminal, the terminal can display, in real time, a cartoon image cartoonized with each real scene image in the video frame sequence after acquiring that the user triggers the cartoon playback mode for the video file. When the cartoon playback mode is not triggered for the video file, that is, in the normal playback mode of the video file, the computer device can directly play the video file itself. It can be understood that this embodiment can implement two playback modes (ie, normal playback mode and cartoon playback mode) for one video file.

When the computer device is the server, when the video file is played on the terminal, the cartoon image corresponding to each real scene image in each video frame sequence in the played video file is output from the server to the terminal in real time, and then the terminal Real-time display of cartoon images.

In one embodiment, the method further includes: when the video frame sequence is an image frame sequence in a real-time video stream, then, in real time, according to the cartoon image of the real scene image in the video frame sequence after cartoonization, Output cartoonized video stream.

It can be understood that when the video frame sequence is an image frame sequence in a real-time video stream, the computer device can perform the methods in the embodiments of the present application on each real scene image in the video stream to convert It is converted into a corresponding cartoon image, and the cartoon image is displayed in real time. In this way, the video stream sending end sends the video stream of the real scene image in real time, and the display end displays the cartoon image of the real scene image in real time, that is, the cartoonized video stream is output in real time.

It can be understood that when the computer device is a terminal, outputting the cartoonized video stream in real time refers to outputting the cartoonized video stream in real time. When the computer device is a server, outputting the cartoonized video stream in real time refers to outputting the cartoonized video stream to the terminal in real time, so that the terminal can display it in real time.

Here are some examples to illustrate. For example, in a live broadcast scene, the live broadcast sender collects the real scene image in the live video stream, and then sends it to the server, and the server performs cartoon processing on it, generates cartoon images, and sends the cartoon images in real time in a streaming manner. To the live broadcast receiving end, so that the real scene where the anchor is located is converted into a cartoon scene and presented to the users watching the live broadcast. For another example, in the video call scene in instant messaging, the real scene images of both parties in the video call can also be converted into cartoon images and displayed in real time in the form of video streams, so that both parties in the video call can see the cartoonized dynamic scene. For another example, when watching videos online on some video content platforms, for an original online video composed of real scene images, after the user selects the cartoon playback mode, the cartoonized video stream can be output in real time, so as to play the real scene images. The corresponding cartoonized image.

In the above embodiment, video files or video streams in real scenes can be converted into cartoonized video files or cartoonized video streams, thereby realizing the diversity of video scenes. In addition, the generated cartoon video file or cartoon video stream has high-quality cartoon images, which improves the picture quality of the video file or video stream.

In one embodiment, performing image reconstruction processing and abstract smoothing processing on the real scene image based on the semantic information of the real scene image to obtain an abstract cartoon image mapped from the real scene image on the cartoon domain includes: inputting the real scene image into the trained image Based on the extracted semantic information, the real scene image is reconstructed, and the reconstructed image content is subjected to bilateral filtering and smoothing to generate abstract cartoon images.

Among them, the deep neural network is used to extract the semantic information of the real scene image, and based on the semantic information, image reconstruction processing and bilateral filtering and smoothing processing are performed. Bilateral filtering and smoothing refers to filtering and abstracting the image from the two dimensions of pixel value and spatial position.

It should be noted that the deep neural network is not limited to being able to implement only the above-mentioned processing, and can also implement other processing.

A deep neural network can be a single neural network or a multi-stage neural network. It can be understood that the multi-stage neural network includes at least two-stage neural networks.

When the deep neural network is a separate neural network, the computer equipment can individually iteratively train the deep neural network according to the sample data set in advance to obtain the final deep neural network. When the deep neural network is a multi-stage neural network, the computer equipment can iteratively train the neural networks of each stage included in the multi-stage neural network, so as to realize the training of the multi-stage deep neural network.

In the above embodiment, through the deep neural network, in the first stage, the key semantic information in the real scene image is deeply understood, and based on the semantic information of the real scene image, image reconstruction processing and abstract smoothing processing are performed on the real scene image. , get the abstract cartoon image of the real scene image mapped on the cartoon domain, and combine the image reconstruction and bilateral filtering smoothing, which not only maintains the saliency structure in the real scene image, but also smoothes out the insignificant details, which improves the Image processing accuracy. Moreover, bilateral filtering can perform filtering abstract smoothing processing on images from the two dimensions of pixel value and spatial position, which improves the accuracy of abstract smoothing processing and image processing quality. Subsequently, high-quality cartoon images can be generated based on the abstract cartoon images generated in the first stage.

In one embodiment, the real scene image is input into the first generator of the abstract network in the deep neural network. In this embodiment, the training steps of the deep neural network include: acquiring a sample data set; the sample data set includes a first set of sample graphs of real scenes; inputting the sample data set into the deep neural network to be trained for iterative training, and in each Determine the structural reconstruction loss value and the bilateral filtering smoothing loss value in the rounds of iterations, and determine the target loss value according to the structural reconstruction loss value and the bilateral filtering smoothing loss value, and in each iteration, adjust the network model parameters according to the target loss value , until the iteration is stopped, and the trained deep neural network is obtained; the network model parameters include the parameters of the first generator; the trained deep neural network includes the trained abstract network.

It can be understood that, in the embodiment of the present application, the deep neural network includes an abstract network, the abstract network includes a first generator, and the computer device inputs real scene images into the first generator to generate abstract cartoon images. That is, the first generator of the abstract network is used to extract and perform image reconstruction and bilateral filtering and smoothing processing based on the semantic information of the real scene image, thereby generating an abstract cartoon image.

The abstract cartoon image may be the final output data of the first generator, or may be an intermediate result generated by the first generator, that is, the first generator may further process based on the abstract cartoon image, and then perform corresponding output.

The deep neural network is a pre-trained neural network. The sample dataset and specific training steps used to train the deep neural network are described below.

The sample data set is the training set used to train the deep neural network. The sample data set includes the first set of real scene sample graphs, but it is not limited to only include the first set, and may also include sets of other sample data. That is, the sample data set may include multiple different types of sample sets, wherein the first set is one type of sample set (ie, a set of real scene sample graphs).

In one embodiment, the first set may include original real scene sample images, or may include real scene sample images that have been preprocessed for the original real scene sample images.

Specifically, the computer device can obtain the original real scene sample image, and then randomly crop the original real scene sample image into an image of a preset size, and then adjust it to a preset resolution, and then resize and The image after resolution adjustment is used as the final real scene sample map, and then forms the first set, which is input to the deep neural network for training.

For example, first collect 5,000 real-world photos (that is, the original real-world sample images). Randomly crop 5000 real-world photos into small pieces of preset size (for example, the size is 500×500), and then adjust the resolution to 256×256, then, the adjusted image is the first set—— That is, a collection of real scene sample graphs. Then, the adjusted images (ie, the first set) can be input into the deep learning network to perform training. It can be understood that by preprocessing the original real scene sample image with size adjustment and resolution adjustment, and then generating the first set, the accuracy of subsequent deep learning network training can be improved, and the data processing pressure can also be reduced.

The target loss value refers to the final loss value in each iteration when the deep neural network is iteratively trained.

In one embodiment, the computer device may obtain the structure reconstruction loss function and the bilateral filtering smoothing loss function that are pre-designed for the abstract network, and then construct the target loss function according to the structure reconstruction loss function and the bilateral filtering smoothing loss function, and input the sample data set as input Iterative training is performed in the deep neural network to be trained. The network model parameters of the deep neural network are adjusted in each iteration to minimize the value of the objective loss function, until the iteration is stopped, and the trained deep learning network is obtained.

It can be understood that in each iteration, the computer equipment can calculate the structural reconstruction loss value and the bilateral filtering smoothing loss value according to the structural reconstruction loss function and the bilateral filtering smoothing loss function in the target loss function, and according to the structural reconstruction loss value and bilateral filtering loss value Smooth the loss value, determine the target loss value corresponding to each iteration, and adjust the network model parameters of the deep neural network according to the target loss value to find the minimum target loss value, so that the deep neural network gradually converges until it stops Iterate to get the trained deep learning network.

It should be noted that the supervised iterative training is performed according to the structural reconstruction loss function, and the unsupervised iterative training is performed according to the bilateral filtering smoothing loss function. Namely, image content reconstruction and abstract smoothing are performed using a supervised structure reconstruction loss and an unsupervised bilateral filtering smoothing loss.

It can be understood that since the deep neural network includes an abstract network, the training of the abstract network must be involved in the training process of the deep neural network. When adjusting the parameters of the network model, the parameters of the first generator must be involved. adjustment. Therefore, the trained deep neural network includes the trained abstract network.

It should be noted that the target loss value is not limited to be determined only according to the structural reconstruction loss value and the bilateral filtering smoothing loss value. When the deep neural network also includes other networks or has other processing, the target loss value can also be determined jointly with other loss values.

Wherein, the structural reconstruction loss value is used to represent the structural feature difference between the cartoon image generated by the first generator according to the real scene sample image and the real scene sample image. It can be understood that the structural reconstruction loss function is considered when constructing the target loss function, in order to limit the structural similarity between the input real scene image and the cartoon image output by the first generator, that is, to make the cartoon image generated by the first generator. , preserve the saliency structure in the input sample map of the real scene (i.e. preserve key structural features).

In one embodiment, the structural reconstruction loss function designed for the abstract network is as follows:

L _ssc =∑(1+ρB)||G(I)-I|| ₂ ;

Among them, L _ssc is the structure reconstruction loss function; G(I) represents the cartoon image (ie the predicted abstract cartoon image) generated by the first generator. I represents the input real scene sample map. B represents a binary mask, in which 1 represents the structural edge, otherwise it is 0, and ρ is the weight of the salient edge. It can be understood that the binary mask can be generated by using a pre-trained line tracking network, that is, the structural edge lines of the sample image of the real scene are extracted through the line tracking network, and then the obtained line image is converted into a binary mask.

Bilateral filtering smoothes the loss value to determine the difference between adjacent pixels in the cartoon image generated by the first generator based on pixel value similarity and spatial position similarity. It can be understood that the bilateral filtering smoothing loss function is considered when constructing the target loss function, in order to comprehensively consider the similarity of pixel values and the similarity of spatial positions to abstract and smooth the image. That is, from the two dimensions of pixel value and spatial position, the image is filtered abstractly smoothed.

Specifically, the computer device may obtain a pre-designed bilateral filtering smoothing loss function. Then, in each iterative process, the bilateral filtering smoothing loss value can be obtained by comprehensively considering the two aspects of pixel value similarity and spatial position similarity in combination with the bilateral filtering smoothing loss function. It can be understood that the bilateral filtering smoothing loss value can be used to characterize the difference between each pixel and its adjacent pixels in the cartoon image generated by the first generator. In this way, the difference between each pixel and its adjacent pixels in the cartoon image generated by the first generator is determined by comprehensively considering the similarity of the pixel value and the similarity of the spatial position.

In one embodiment, the bilateral filtering smoothes the loss function, including two kernel functions, a spatial kernel and a pixel range kernel. That is, the final weight coefficient function of the bilateral filtering smoothing loss function is determined by the spatial kernel and the pixel range kernel. Among them, the spatial kernel is the weight coefficient function of the spatial domain, which is used to smooth the size of the spatial neighborhood of the pixels, that is, the weight coefficient is determined from the spatial position similarity. The pixel range kernel is the weight coefficient function of the pixel range domain, which is used to smooth the pixel value (color difference) size between adjacent pixels, that is, to determine the weight coefficient from the similarity of pixel values. Therefore, the final weight coefficient function of the bilateral filtering smoothing loss function is a comprehensive weight coefficient function that considers the similarity of pixel values and the similarity of spatial positions. Therefore, when using the bilateral filtering smoothing loss function to calculate the bilateral filtering smoothing loss value, it can be comprehensively considered from the two aspects of pixel value similarity and spatial position similarity.

In one embodiment, the formula of the bilateral filtering smoothing loss function _Lfla is as follows:

定义为

表示空间域的权重系数函数；

定义为

表示像素范围域的权重系数函数。σ表示高斯核函数的标准差。σ_co表示空间内核函数的标准差，σ_sp表示像素范围内核的标准差。C表示颜色通道，x和y表示相素的空间位置坐标。x_i,y_i则为像素i的空间位置坐标，x_j,y_j则为像素j的空间位置坐标。I表示输入的真实场景样本图。I_i,c-I_j,c表示I中的像素i和其相邻像素j之间的像素值之差。G(I)_i表示第一生成器生成的卡通图像中的像素i，G(I)_j表示第一生成器生成的卡通图像中的像素j。Among them, L _fla is the smoothing loss function of bilateral filtering; N is the total number of pixels, and N _S (i) represents the set of adjacent pixels of pixel i in the H×H range. j represents an adjacent pixel of pixel i. |·| ^z means L2 regularization operation without square root. in

defined as

represents the weight coefficient function of the spatial domain;

defined as

Represents a weight coefficient function for the pixel range domain. σ represents the standard deviation of the Gaussian kernel function. σ _co represents the standard deviation of the spatial kernel function, and σ _sp represents the standard deviation of the pixel-wide kernel. C represents the color channel, and x and y represent the spatial position coordinates of the pixels. x _i , y _i are the spatial position coordinates of pixel i, and x _j , y _j are the spatial position coordinates of pixel j. I represents the input real scene sample map. I _i,c -I _j,c represents the difference in pixel value between pixel i in I and its neighbor pixel j. G(I) _i represents pixel i in the cartoon image generated by the first generator, and G(I) _j represents pixel j in the cartoon image generated by the first generator.

In one embodiment, σ _co and σ _sp may be set to 0.2 and 5, respectively.

In the above embodiment, the supervised structural reconstruction loss and the unsupervised bilateral filtering smoothing loss are combined to perform image content reconstruction and abstract smoothing training to obtain a deep neural network, which can then be based on the trained deep neural network. , perform high-quality image content reconstruction and abstract smoothing, thereby improving the quality of subsequent image generation.

In one embodiment, stylizing the abstract cartoon image to generate a style cartoon image with an artistic style includes: in a first stage, performing stylization processing on the abstract cartoon image by a first generator to generate an artistic style cartoon image. style cartoon image.

It can be understood that, in addition to generating the abstract cartoon image, the first generator can further stylize the abstract cartoon image to generate a style cartoon image with artistic style.

It should be noted that, in this embodiment, since the first generator can perform stylization processing, stylization-related training must also be performed in the training process. That is, the abstract network is pre-trained for style transfer so that its first generator can be stylized.

In one embodiment, the abstraction network further includes a first discriminator. In this embodiment of the present application, the abstract network is a generative adversarial network (GAN model) including a first generator and a first discriminator. Therefore, the network model parameters of the deep neural network also include the parameters of the first discriminator.

In this embodiment, the sample data set further includes a second set of abstract cartoon sample images. Among them, the abstract cartoon sample image is an abstract image with missing outline edge lines. Understandably, the abstract cartoon sample images in the second collection all have the same artistic style.

It can be understood that the purpose of the first discriminator is to distinguish the cartoon image generated by the first generator from the style of the abstract cartoon sample image that lacks contour edge lines. The first generator and the first discriminator are antagonistic, so it is necessary to iteratively generate cartoon images that the first discriminator cannot distinguish, so that the styles of the generated cartoon images and the abstract cartoon sample images are gradually approached, so as to achieve The style features of the abstract cartoon sample images are introduced into the cartoon images generated by the first generator.

In one embodiment, the structural reconstruction loss value and the bilateral filtering smoothing loss value are determined in each round of iteration, and according to the structural reconstruction loss value and the bilateral filtering smoothing loss value, determining the target loss value includes: in each round of iteration, A structure reconstruction loss value, a bilateral filter smoothing loss value and a style enhancement loss value are determined, and a target loss value is determined according to the structure reconstruction loss value, the bilateral filter smoothing loss value and the style enhancement loss value.

Specifically, the computer device may obtain a pre-designed style enhancement loss function, and construct the target loss function according to the structural reconstruction loss function, the bilateral filtering smoothing loss function, and the style enhancement loss function. It can be understood that the purpose of considering the style enhancement loss function when constructing the target loss function is to make the cartoon images generated by the first generator have the style of abstract cartoon sample images, that is, to enable the first generator to generate the style of abstract cartoon sample images. cartoon image.

When training the deep neural network, the computer equipment may input the sample data sets including the first set and the second set into the deep neural network for iterative training, and in each iteration, randomly select a sample data set from the first set Sample images of real scenes, and randomly select a preset number (for example, 5) of abstract cartoon sample images from the second set as a reference, and combine with the target loss function to obtain the structural reconstruction loss value corresponding to this iteration, the bilateral Filter the smoothing loss value and the style enhancement loss value, and then determine the final target loss value according to the structural reconstruction loss value, the bilateral filtering smoothing loss value and the style enhancement loss value, so as to iteratively adjust the network model parameters according to the target loss value. The model parameters include the parameters of the first generator and the parameters of the first discriminator until the iteration stops. Likewise, the target loss value is not limited and may also include other loss values. That is, the target loss function may also include other loss functions.

It can be understood that the style enhancement loss value is determined by the first discriminator according to the first probability output by the cartoon image generated by the first generator. That is, in each iteration, the cartoon image generated by the first generator in this iteration can be input to the first discriminator, and the first discriminator can output a probability value according to the input cartoon image—that is, the first probability, and then determine the style enhancement loss value according to the first probability. The first probability is used to represent the probability that the cartoon image generated by the first generator belongs to the style of the abstract cartoon sample image. It can be understood that the larger the first probability, the closer the cartoon image generated by the first generator is to the abstract cartoon sample image, and the harder it is for the first discriminator to recognize.

In one embodiment, the adversarial loss function (L _sty-D ) of the first discriminator is formulated as follows:

L _sty-D =∑log D(A _y )+log(1-D(G(I)));

The formula for the style enhancement loss function L _sty of the first generator is as follows:

Among them, A _y ={a _y |y=1,...,Y}∈A. A is the second set, and A _y is the Y-th abstract cartoon sample image. I and G(I) represent the input real scene sample graph and the cartoon image generated by the first generator, respectively. D is the first discriminator; D( ) is the output result of the first discriminator; D(A _y ) is the output result after the Y-th abstract cartoon sample image A _y is input to the first discriminator, D(G( 1)) is the first probability output after inputting the cartoon image generated by the first generator to the first discriminator.

It should be noted that the loss value of structural reconstruction is determined in a supervised way, and the loss value of bilateral filtering and style enhancement loss are determined in an unsupervised way, so as to reconstruct the loss value based on supervised structure and unsupervised bilateral filtering. The loss value is smoothed, image content reconstruction and abstract smoothing are performed, and stylized transfer is performed using generative adversarial loss in an unsupervised manner, thereby generating abstract smooth high-quality stylized cartoon images in the first stage.

In one embodiment, the step of obtaining the second set includes: obtaining a third set of original cartoon images; extracting contour edge lines of each original cartoon image according to a pre-trained line tracing network; according to the extracted contour edge lines and the original cartoon image image, generate an abstract cartoon sample image, and obtain the second set; the abstract cartoon sample image is an abstract image after removing the outline edge lines from the original cartoon image.

Among them, the original cartoon images in the third collection have the same artistic style. Well, the abstract cartoon sample images in the second set also have the same artistic style as the original cartoon images in the third set.

It should be noted that the original cartoon images of the required artistic style can be prepared as required to obtain a third set, and then based on the third set, abstract cartoon sample images belonging to the artistic style can be obtained, and used as the second set for training, thereby It can flexibly generate cartoon images in the desired artistic style. That is, according to the training data without artistic styles, cartoon images with different artistic styles can be generated.

For example, if you want to generate A style, you can prepare the third set of A style, so as to get the second set of A style for training, then the trained first generator can convert real scene images into A style cartoons image. Assuming that you want to generate B style, you can prepare the third set of B style, so as to obtain the second set of B style for training, then the trained first generator can convert real scene images into B style cartoon images.

Line tracing network is a pre-trained deep neural network for extracting edge lines. That is, it is used to extract the edge lines of saliency regions of the image, while ignoring redundant lines of non-salient regions.

In one embodiment, the line tracking network, which may be a fully convolutional deep neural network, includes an encoder and a decoder.

Specifically, a line tracing network can be pre-trained in the computer, and then, a third set of preset original cartoon images is input into the line tracing network, and the line tracing network extracts the contour edge lines of each original cartoon image in the third set . Computer equipment can convert this line image into a binary mask. All edge pixels in this binary mask are assigned 1, otherwise 0. Further, the computer device may use a Gaussian filter (such as a Gaussian filter with a standard layer of 5) to generate a Gaussian blurred version of the cartoon image. The computer device can then multiply the Gaussian blurred version of the cartoon image with the binary mask to get blurred edge regions. The computer device can replace the corresponding edge region in the original cartoon image with the fuzzy edge region, so as to obtain an abstract cartoon sample image with missing edge lines, thereby obtaining the second set. It can be understood that there is a one-to-one correspondence between the original cartoon image and the converted abstract cartoon sample image.

In one embodiment, the training step of the line tracking network includes: acquiring a line tracking data set; the line tracking data set includes multiple sets of cartoon line image pairs, and each set of cartoon line image pairs includes a sample cartoon image and a line of each sample cartoon image Figure; According to the line tracking dataset, supervised iterative training of the line tracking network to obtain the final line tracking network.

It can be understood that the line tracing network can be supervised using the L1 norm loss function. To enhance data diversity and tolerance to background regions, some images with cluttered backgrounds can be included in the line-tracing dataset, allowing the line-tracing network to focus on important structural edges. By changing the line tracing network, the outline and edge lines of cartoon images of various styles can be extracted.

In the above embodiment, through the pre-trained line tracing network, the contour edge lines of each original cartoon image can be accurately extracted, so that more reference-valued abstract cartoon sample images can be generated according to the extracted contour edge lines and the original cartoon image. Then, based on the abstract cartoon sample image, a more accurate deep neural network can be trained, and then a higher quality cartoon image can be generated based on the deep neural network.

In one embodiment, the deep learning network is a two-stage neural network, and the abstract network is a first-stage abstract network in the two-stage neural network; the two-stage neural network further includes a second-stage line drawing network. In this embodiment, step 206 generates the outline edge lines of the style cartoon image, and obtaining the cartoon image after the real scene image is cartoonized includes: in the second stage, inputting the style cartoon image generated in the first stage into the trained A second generator in the second-stage line drawing network in the two-stage neural network, so as to perform contour edge line generation processing on the style cartoon image through the second generator, so as to obtain a cartoon image after the cartoon image of the real scene image; wherein, the second The stage line drawing network is a deep neural network that generates contour edge lines in the second stage.

Among them, the two-stage neural network includes a two-stage neural network. It can be understood that the neural networks in different stages are used for different processing, and the output result of the neural network in the previous stage is used as the input data of the neural network in the subsequent stage. It should be noted that, in the training process of the two-stage neural network, the input of different stages may be different or the same type of sample sets in the sample data set.

Two-stage neural network, including a first-stage abstraction network and a second-stage line drawing network. The first-stage abstract network refers to a deep neural network that performs abstract smoothing and stylization processing in the first stage. The second-stage line delineation network is a deep neural network that describes the contour edge lines for the images output in the first stage. That is, the image output in the first stage is input to the second generator in the line drawing network in the second stage, so that the outline edge line generation processing is performed on the style cartoon image by the second generator, so as to output the real scene image after cartoonization of the final cartoon image.

In the above embodiment, based on a two-stage neural network, the image is abstracted, smoothed and stylized (that is, performed in the first stage), and the edge generation processing (that is, performed in the second stage) is performed in stages, which is the output of the first stage. The addition of contour edge lines avoids the generation of lines in areas that need to be smoothed, and achieves a harmonious unity between the sense of line and the smoothness of image content.

In one embodiment, the abstract cartoon sample image is an abstract image in which outline edge lines are eliminated from the original cartoon image; the sample data set further includes a third set of original cartoon images corresponding to the abstract cartoon sample image. In this embodiment, in each iteration, the structural reconstruction loss value, the bilateral filtering smoothing loss value, and the style enhancement loss value are determined, and the target is determined according to the structural reconstruction loss value, the bilateral filtering smoothing loss value, and the style enhancement loss value. The loss values include: in each iteration, determine the structure reconstruction loss value, bilateral filter smoothing loss value, style enhancement loss value and edge line distribution loss value, and according to the structure reconstruction loss value, bilateral filter smoothing loss value, style enhancement value The loss value and the edge line assign the loss value to determine the target loss value.

It can be understood that the computer device can obtain a pre-designed edge line assignment loss function. That is, the objective loss function also includes the edge line assignment loss function.

Specifically, in each round of iterative training, the computer device may input the abstract cartoon sample image into the second generator of the line description network to be trained, so as to perform supervised iterative training on the line description network, in each iteration , according to the edge line distribution loss function, after the abstract cartoon sample image is input to the second generator, the difference between the image with lines generated by the second generator and the original cartoon image corresponding to the abstract cartoon sample image is determined , so as to get the edge line distribution loss value. That is, the edge line assignment loss value is used to represent the difference between the image with lines generated when the abstract cartoon sample image is used as the input of the second generator, and the original cartoon image corresponding to the abstract cartoon sample image.

The computer device may determine the target loss value corresponding to the current iteration according to the loss value of structural reconstruction, the loss value of bilateral filtering, the loss value of style enhancement, and the loss value of edge line distribution. Thereby, the network model parameters of the two-stage neural network are adjusted according to the target loss value. The network model parameters further include parameters corresponding to the first generator, the first discriminator, and the second generator.

In one embodiment, the formula of the edge line assignment loss function _Leas is as follows:

Wherein, L(A _y ) represents the image with lines generated by the second generator after taking the abstract cartoon sample image A _y as input; C _GT represents the original cartoon image corresponding to the abstract cartoon sample image A _y .

It should be noted that only in the training phase, will it involve the line drawing of abstract cartoon sample images, so as to conduct supervised iterative training. When the line drawing network is trained and used, only the image output by the first generator in the first stage is used as input, and there is no line drawing for the abstract cartoon sample image.

In the above embodiment, supervised line drawing training is performed in the second stage, and the second stage line drawing network for generating contour edge lines can be accurately trained, so that when the network model is used, it can be based on the second stage line. The result of the first stage output by the delineation network accurately outlines the edge lines, realizes the line feeling of the cartoon image, and avoids the generation of cluttered and redundant lines.

In one embodiment, the second-stage line drawing network further includes a second discriminator; the network model parameters further include parameters of the second discriminator.

It can be understood that, in this embodiment, the second-stage line drawing network is a generative adversarial network including a second generator and a second discriminator. Then, the network model parameters of the two-stage neural network also include the parameters of the second discriminator.

The purpose of the second discriminator is to distinguish the image with lines generated by the second generator from the line intensity of the original cartoon image. While the second generator and the second discriminator are antagonistic, it is necessary to iteratively generate an image with lines that the second discriminator cannot distinguish, so that the line intensity of the generated image and the corresponding original cartoon image gradually approaches, This enables enhancements to the generated lines.

It can be understood that the image with lines generated by the second generator and the abstract cartoon sample image with real outline and edge lines are all fake sample data, while the original cartoon image is real sample data. Thereby, the second generator is guided to generate a line-enhanced image that is indistinguishable by the second discriminator.

Wherein, when the second generator performs adversarial training with the second discriminator, the images output by the first generator are input into the second generator for unsupervised iterative training.

In one embodiment, in each iteration, a structure reconstruction loss value, a bilateral filter smoothing loss value, a style enhancement loss value, and an edge line assignment loss value are determined, and according to the structure reconstruction loss value, bilateral filter smoothing loss value, The style enhancement loss value and the edge line distribution loss value determine the target loss value, including: in each iteration, determine the structure reconstruction loss value, the bilateral filter smoothing loss value, the style enhancement loss value, the edge line distribution loss value and the edge line enhancement value Loss value; determine the target loss value according to the structural reconstruction loss value, the bilateral filtering smoothing loss value, the style enhancement loss value, the edge line distribution loss value and the edge line enhancement loss value.

Specifically, the computer device can obtain the pre-designed edge line enhancement loss function, and construct the target loss function according to the structure reconstruction loss function, the bilateral filtering smoothing loss function, the style enhancement loss function, the edge line assignment loss function, and the edge line enhancement loss function. It can be understood that the edge line enhancement loss function is considered when constructing the target loss function, so that the line intensity of the image generated by the second generator is close to the line intensity of the original cartoon image.

Then, when training the two-stage neural network, the sample dataset is input into the two-stage neural network for iterative training. In each iteration, the computer device can combine the target loss function to obtain the structure reconstruction loss value, the bilateral filtering smoothing loss value, the style enhancement loss value, the edge line assignment loss value and the edge line enhancement loss value corresponding to the current iteration , and then obtain the final target loss value, so as to iteratively adjust the network model parameters according to the target loss value until the iteration stops.

It can be understood that the loss value of edge line enhancement is the image with lines generated when the second generator takes the image generated by the first generator as input, and after inputting it to the second discriminator, the output of the second discriminator is The second probability is determined; the second probability is used to characterize the line intensity difference between the image with lines generated by the second generator and the original cartoon image. That is, in each iteration, the image generated by the first generator can be input to the second generator, and the image with lines generated by the second generator according to the input image can be input to the second discriminator, The second discriminator outputs a probability value—that is, the second probability. Further, the edge line enhancement loss value is determined according to the second probability. It can be understood that the larger the second probability, the closer the line intensity of the image generated by the second generator is to the original cartoon image, and the more difficult it is for the second discriminator to recognize, so as to realize the enhancement processing of contour edge lines.

In one embodiment, the edge enhancement adversarial loss function ( _Leau-D ) is formulated as follows:

In one embodiment, the formula of the edge line enhancement loss function L _eau corresponding to the generator is as follows:

Among them, A _y is the abstract cartoon sample image, C _GT represents the original cartoon image corresponding to the abstract cartoon sample image A _y ; _DL represents the second discriminator, _DL ( ) represents the result output by the second discriminator, G(I ) and L(G(I)) denote the images generated by the first generator of the first stage and the images generated by the second generator of the second stage, respectively. D _L (L(G(I))) represents the second probability output after the image generated by the second generator is input to the second discriminator. DL ( _{C GT} ) represents the output result after inputting the original cartoon image to the second discriminator; DL _{(A y ) represents the output result after inputting the abstract cartoon sample image A y to the} second discriminator.

In one embodiment, the formula of the final objective loss function L _all is as follows:

Among them, L _ssc is the structure reconstruction loss function, L _sty is the style enhancement loss function, L _fla is the bilateral filtering smoothing loss function, L _eas is the edge line allocation loss function, and L _eau is the edge line enhancement loss function. α, β, γ, δ, θ are parameters. In one embodiment, α, β, γ, δ, θ can be heuristically set to 10, 2, 1.5, 50, and 1, respectively.

That is, in the second stage, the edge line assignment loss value is determined in a supervised manner, and the edge line enhancement loss value is determined in an unsupervised manner, thereby in the second stage by supervised edge line assignment and unsupervised edge line enhancement During training, the edge contour lines are drawn on the basis of the cartoon images generated in the first stage, which reduces the generation of unnecessary lines and improves the quality of the final cartoon images.

In order to facilitate the understanding of the two-stage neural network, a schematic illustration of the network structure of the two-stage neural network is now given with reference to FIG. 3 . The two-stage neural network includes a first-stage abstraction network and a second-stage line drawing network. FIG. 3 mainly shows the structures of the first generator and the second generator, and does not show the structures of the first discriminator and the second discriminator.

Referring to Fig. 3, the first generator G in the first stage abstraction network is a fully convolutional network with the structure of encoder E ₁ and decoder D ₁ . The encoder E ₁ and the decoder D ₁ may be sequentially stacked by two convolutional blocks. Each convolution block contains a convolutional layer with a kernel size of 3 × 3, an instance normalization layer, and a ReLu (linear rectification function) activation layer. Rich features are extracted in the encoding convolutional layers, and the resolution is upscaled to the original resolution during decoding. It can be understood that the detailed structure of each convolution block is not shown. In the middle of the network, several residual blocks R1 (eg, ₉ ) are added to supplement useful functions and eliminate artifacts.

Further, in order to make full use of the original input, a single-step translation block (Flat Conv) can be set in front of the encoder E1. The translation block F _1e can be _set with 64 convolution kernels of 7 × 7 size, thereby expanding Receptive fields and reintegrate features to learn more global representations. A flat transform block F _1d can be set after the decoder D ₁ to transform the features output by the decoder D ₁ into a three-channel output. Reduces artifacts at surrounding image boundaries with Reflection ping. At the end of the first generator G, a skip residual connection from input to output is adopted, followed by a Tanh (one of the hyperbolic function) activation layer T to obtain the abstract result finally generated by the first generator - that is, the generated abstract cartoon image.

The first discriminator in the first-stage abstraction network, consisting of three convolutional blocks with a kernel size of 4 × 4. After the normalization layer of each convolutional block, Leaky ReLU (linear unit function with leaky correction) is used as the activation function. The final classification result can be obtained by using a convolution kernel to convert the output features into binary values. It can be understood that for the first discriminator, it is equivalent to apply a block-level patch-level structure to focus on local feature learning. The figure does not show the structure of the first discriminator.

The lines in the second stage depict the structural composition of the network, which is similar to the structural composition of the abstract network in the first stage. The structure of the second discriminator adopts the structure of the first discriminator (the structure of the second discriminator is also not shown in FIG. 3 ), and the second generator L adopts a structure similar to that of the first generator G.

It can be seen from FIG. 3 that the second generator L also includes an encoder E ₂ and a decoder D ₂ . Before the decoder D ₂ and after the encoder E ₂ , there are flat blocks F _2d and F _2e respectively, and the encoder E ₂ There are several residual blocks R _{2 between it and the decoder D 2 .} The second generator L differs from the first generator G in two ways. The first difference is that the number of residual blocks used by the second generator is smaller than the number of residual blocks in the first generator (eg, if the first generator uses 9 residual blocks, the second generator can use 6 residual blocks). Because the edge line information is simpler than the information contained in the color style, shallow networks are more suitable, so the second generator can use a smaller number of residual blocks. The second difference is that the second generator does not employ jump connections from input to output compared to the first generator. It should be noted that FIG. 3 only shows the network structure of the second generator L, which does not mean that the output result G(I) of the first stage is input to F _2d , and then F _2e outputs the final result, but The output result G(I) of the first stage is first passed through the encoder in the second generator L, and then passed through the decoder of the second generator L after encoding, and the final output is L(G(I)), that is, sequentially Go through F _2e —>E ₂ —>R ₂ —>D ₂ —>F _2d , thereby outputting L(G(I)).

With the network structure shown in FIG. 3 , a brief flow of the cartoonization processing of the real scene will now be described.

First, the brief flow of the training process of the two-stage network is introduced. In the training process, the sample data set includes a first set P of real scene sample images, a second set A of abstract cartoon sample images, and a third set C of original cartoon images corresponding to each abstract cartoon sample image. Then, in each iterative training, the first set is input into the first-stage abstract network to be trained, and the first generator G performs image reconstruction processing and bilateral filtering and smoothing processing, and outputs an abstract cartoon image G(I) . Among them, in the training phase, I represents a real scene sample map in the first set P. Therefore, the structural reconstruction loss value of this training can be determined according to the structural difference between the cartoon image G(I) and the real scene sample image I, and the cartoon image G( The difference between adjacent pixels in I), resulting in a bilateral filtering smoothing loss value.

The first discriminator can randomly select a preset number of abstract cartoon sample images from the second set A as a reference to determine the probability that the output cartoon image G(I) of the first generator G belongs to the style of the abstract cartoon sample image (that is, the first a probability), so as to calculate the style enhancement loss value. Shown in Figure 3 are 5 abstract cartoon sample images (such as the 5 images shown in dotted box 302).

In each iteration of training, the training of the line drawing network for the second stage can start with an abstract cartoon sample image in the second set (eg, _Ay shown in Figure 3) as input, and from the third The original cartoon image C _GT corresponding to the abstract cartoon sample image A _y is found in the collection as a reference, and supervised training is performed according to the abstract cartoon sample image A _y and its corresponding original cartoon image C _GT , and the loss value of edge line assignment is obtained.

In addition, in each iteration of training, the G(I) output in the first stage can also be input into the second stage line drawing network, and the second generator can draw lines for it, thereby generating cartoon images with lines L(G(I)). The cartoon image L(G(I)) generated by the second generator is input to the second discriminator, and the second discriminator can use the original cartoon image C _GT as a reference, and output a cartoon image L(G(I) for characterizing the cartoon image L(G(I) ) ) and the original cartoon image C _GT of the line intensity difference probability (ie the second probability), so as to obtain the edge line enhancement loss value.

Then, the target loss value is determined according to the structure reconstruction loss value, the bilateral filter smoothing loss value, the style enhancement loss value, the edge line assignment loss value, and the edge line enhancement loss value in this training. Therefore, according to the target loss value, the parameters of the first generator, the first discriminator, the second generator and the second discriminator are adjusted, and then enter the next iterative training, and so on, until the training is stopped, and the training is completed. A two-stage neural network.

After the two-stage neural network is trained, when using it to cartoonize the real scene images, only the generator in the first-stage abstract network and the second-stage line drawing network are used for processing, and the discriminator is only used in the training process. middle.

The input and output flow in the use of the two-stage neural network will now be briefly described. Input the real scene image I (it can be understood that when the network is used, I represents the real scene image to be cartoonized) into the first generator G to generate an abstract cartoon image, which is processed by the first generator G. Stylized processing to generate a style abstract image with artistic style (it can be understood that when the network is used, G(I) is the abstract image with artistic style output by the first generator G—the style abstract image), and then, the The style abstract image is input into the second generator L, and the final cartoon image is output (in the case of network use, L(G(I)) is the final cartoon image output by the second generator).

In one embodiment, the iterative training includes structural reconstruction iterative training and network stacking iterative training.

It can be understood that the entire two-stage neural network can be iteratively trained in an end-to-end manner, ie, the first-stage abstraction network and the second-stage line drawing network can be iteratively trained simultaneously. In addition, iterative training can also be split into structure reconstruction iterative training and network stacking iterative training, which can speed up the training process and improve convergence.

Among them, the structure reconstruction iterative training is used to train the initialization abstract network for image reconstruction processing. Network layered iterative training refers to layering the initial abstract network and the second-stage line drawing network for iterative training. That is, the first-stage abstract network is initialized and trained through structural reconstruction training to obtain the initialized abstract network. The initialized abstract network only has the function of image reconstruction and cannot achieve abstract smoothing. Therefore, the initialized abstract network and the second-stage line drawing network can be layered together for iterative training, so as to generate images that can achieve image reconstruction and abstract smoothing. A first-stage abstraction network and a second-stage line-drawing network capable of generating lines.

In this embodiment, the sample data set includes a first set of real scene sample images, a second set of abstract cartoon sample images, and a third set of original cartoon images corresponding to the abstract cartoon sample images. The training steps of the deep neural network include: inputting the first set into the abstract network to be trained for structural reconstruction iterative training, determining the structural reconstruction loss value in each round of structural reconstruction iterative training, and adjusting the first set according to the structural reconstruction loss value. The parameters of the generator until the initial training stop condition is reached, and the initialized abstract network is obtained; the first set and the second set are input to the initialization abstract network, and the second set and the third set are input to the second stage to be trained. Line drawing The network performs network cascade iterative training, and in each iteration, determines the loss value of structure reconstruction, the loss value of bilateral filter smoothing, the loss value of style enhancement, and the loss value of edge line distribution; according to the loss value of structure reconstruction, the loss value of bilateral filter smoothing , the style enhancement loss value, and the edge line distribution loss value to determine the comprehensive loss value; according to the comprehensive loss value, adjust the parameters of the first generator and the first discriminator to initialize the abstract network, and adjust the line drawing of the second stage to be trained The parameters of the second generator and the second discriminator of the network are used until the iteration stops, and the trained two-stage neural network is obtained; the two-stage neural network includes the first-stage abstract network and the second-stage line drawing network.

Specifically, first, the computer device may pre-train an initialization abstraction network for image reconstruction. That is, the computer device can input the first set into the abstract network to be trained for structural reconstruction iterative training, determine the structural reconstruction loss value in each round of structural reconstruction iterative training, and adjust the parameters of the first generator according to the structural reconstruction loss value , until the initial training stop condition is reached, and the initialized abstract network is obtained. In this way, the resulting initialized abstract network can be used for image reconstruction to generate basic abstract images. In one embodiment, when the number of iterations of the structural reconstruction iterative training is relatively small (for example, 10 iterations), the initialization abstract network can be trained.

Then, the computer device can input the first set and the second set to the initialization abstract network, and input the second set and the third set to the second-stage line drawing network to be trained for network stacking iterative training, and at each iteration Determine the comprehensive loss value in , and according to the comprehensive loss value, adjust the parameters of the first generator and the first discriminator to initialize the abstract network, and adjust the second generator and the second discriminator of the line drawing network in the second stage to be trained parameters until the iteration stops, and the trained two-stage neural network is obtained; the two-stage neural network includes the first-stage abstract network and the second-stage line drawing network; among them, the comprehensive loss value is calculated by the network layered and iteratively trained for each round of training. The structure reconstruction loss value, the bilateral filter smoothing loss value, the style enhancement loss value, and the edge line distribution loss value determined in the iteration are determined; the comprehensive loss value is the target loss value determined in each iteration of the network stacking iterative training. Equivalently, in each iteration of the network stacking iterative training, the loss value of structure reconstruction, bilateral filter smoothing loss value, style enhancement loss value, and edge line distribution loss value are determined, and according to the structure reconstruction loss value, bilateral filter smoothing loss value value, style enhancement loss value, and edge line assignment loss value, determine the target loss value in each iteration, to adjust the model parameters of the initial abstract network and the line drawing network of the second stage to be trained, so that the network stacks iteratively. Training is also performed on structural reconstruction, bilateral filter smoothing, style enhancement, and edge line assignment.

It can be understood that the loss value of the edge line distribution is based on the abstract cartoon sample image in the second set as the input, and the corresponding original cartoon image is used as the output reference to perform supervised training to obtain the loss value, so as to supervised training of the first. Two-stage lines delineate the network.

In one embodiment, the composite loss value may also include an edge line enhancement loss value. It can be understood that the loss value of edge line enhancement is to enable the second-stage line drawing network to generate enhanced and clear outline edge lines through unsupervised training. Equivalently, the training of structure reconstruction, bilateral filter smoothing, style enhancement, edge line assignment, and edge line enhancement is performed in the network cascade iterative training.

In one embodiment, the method further includes: in the iterative training process of the network stacking, when one of the initialized abstract network and the second-stage line delineation network to be trained reaches the training stop condition, and the other network does not reach the training stop When the training stop condition is reached, the network model parameters of the network that has reached the training stop condition are stopped, and the training of another network that has not reached the training stop condition is continued until the training stop condition is reached.

For example, after about 100 rounds of training, a satisfactory first-stage abstract network can be generated, and the training of the first-stage abstract network can be stopped. For the second-stage line drawing network, it takes 200 epochs of training to achieve the effect. Therefore, after stopping the training of the first-stage abstract network, you can continue to train the second-stage line drawing network until the training stop condition is reached.

In the above embodiment, an initialization abstract network for realizing image reconstruction is trained first, and then the initialization abstract network and the second-stage line drawing network are layered together for iterative training, so that the convergence can be performed more quickly, and an image that can realize the image can be obtained. A first-stage abstraction network for reconstruction and abstraction smoothing and a second-stage line drawing network capable of generating lines.

It should be understood that although the steps in the above flow charts are displayed in sequence according to the arrows, these steps are not necessarily executed in the sequence indicated by the arrows. Unless explicitly stated herein, the execution of these steps is not strictly limited to the order, and these steps may be performed in other orders. Moreover, at least a part of the steps in the above flow chart may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but may be executed at different times. The execution sequence of these steps or stages It is also not necessarily performed sequentially, but may be performed alternately or alternately with other steps or at least a portion of a step or phase within the other steps.

In order to prove the effect that the method of the present application can produce, the description will now be made with reference to FIG. 4 to FIG. 7 .

Referring to FIG. 4 , 402 in FIG. 4 is an input real scene image, and 404 to 408 are cartoon images of different styles generated from the input real scene image. It can be understood that deep learning networks trained with different styles of sample datasets (ie, abstract cartoon sample images of different styles or original cartoon images) can be used to generate cartoon images of different styles. For example, the deep learning network used to generate 404 cartoon images in artist A's style uses abstract cartoon sample images or original cartoon images belonging to artist A's style as the sample data set during training. For another example, the deep learning network used to generate cartoon images in the style of C artist 408 uses abstract cartoon sample images or original cartoon images in the style of C artist as the sample data set during training.

Figure 5 is used to show the results in different states. Referring to FIG. 5 , (a) is the input real scene image, and (b) to (e) are the intermediate results. (f) is the final result, that is, the final cartoonized cartoon image. (f) Compared with (b) to (e), the cartoon images are of higher quality and more obvious artistic style.

FIG. 6 is a comparison experiment effect diagram of the present case and the reference group in one embodiment. (a) in Figure 6 is the input real scene image, (b) is the cartoon effect of CycleGAN (CycleGAN is essentially two mirror-symmetrical GANs, forming a ring network, which can realize cyclic style conversion); ( c) The cartoon effect of CartoonGAN (CartoonGAN, which is a neural network model based on a GAN model to convert real photos into cartoon images) in CVPR (IEEE International Conference on Computer Vision and Pattern Recognition) 2018; (d) CartoonGAN2 (CartoonGAN2 is a fine-tuning (finetuned CartoonGAN training result) cartoon effect; (e) cartoon effect of this scheme. And each group of renderings are three different styles of cartoon images. From top to bottom are the style of artist A, the style of artist B, and the style of artist C. It can be seen from Figure 6 that the cartoon effect (e) of this scheme can more accurately retain the saliency structure. For example, compared to (b) and (d), the salient structures of clouds and stones, as well as character backgrounds are better preserved. Moreover, compared with (b) and (d), the color abstraction is smoother, and the lines are sharper. For example, the lines of the bridge in (b) and (d) are very unclear. Therefore, the cartoon effect of this scheme is higher than that of (b) and (d) cartoon images.

The cartoon effect of this scheme also avoids the generation of lines in areas that need to be smoothed. For example, compared with (c) clouds and ocean waves, it avoids the generation of cluttered lines, which makes the picture smoother, and also avoids the problems of color overflow, structural distortion, and large-scale artifacts in (c). Therefore, the cartoon effect of this solution is higher than the cartoon image quality of (c).

FIG. 7 is a comparison diagram of realizing cartoon images of different styles by adopting the solution of the present application. From left to right are the input real scene graphs, artist A's style, artist B's style, and artist C's style. It can be understood that the method according to this case can generate cartoon images of different artistic styles by using different styles of training data.

In one embodiment, as shown in FIG. 8 , a processing apparatus for cartoonizing a real scene image is provided. The apparatus can use software modules or hardware modules, or a combination of the two to become a part of computer equipment. Including: acquisition module 802, abstract processing module 804 and line generation module 806, wherein:

The acquiring module 802 is used for acquiring real scene images.

The abstract processing module 804 is used to perform image reconstruction processing and abstract smoothing processing on the real scene image based on the semantic information of the real scene image, so as to obtain an abstract cartoon image in which the real scene image is mapped on the cartoon domain; the abstract cartoon image has a real scene The saliency structure in the image and the lack of contour edge lines in the real scene image; stylize the abstract cartoon image to generate a style cartoon image with artistic style.

The line generation module 806 is used for generating outline and edge lines of the style cartoon image, so as to obtain a cartoon image after cartoonization of the real scene image.

In one embodiment, the real scene images are sequentially extracted from a video frame sequence; the video frame sequence includes at least two real scene images arranged in sequence;

The device also includes:

The output module 808 is configured to, when the video frame sequence is an image frame sequence in a video file, generate a cartoonized video file according to the cartoon image of each real scene image in the video frame sequence after cartoonization, or, when the video file is During playback, the cartoon images corresponding to the real scene images in the video frame sequence are acquired in real time and output.

In one embodiment, the output module 808 is further configured to output the cartoonized video stream in real time according to the cartoon image of the real scene image cartoonized in the video frame sequence when the video frame sequence is an image frame sequence in the real-time video stream.

In one embodiment, the abstract processing module 804 is further configured to input the real scene image into the trained deep neural network, so that in the first stage, semantic extraction is performed on the real scene image, and based on the extracted semantic information, the real scene image is extracted. The image is subjected to image reconstruction processing, and the reconstructed image content is subjected to bilateral filtering and smoothing processing to generate an abstract cartoon image.

In one embodiment, the real scene image is input into the first generator of the abstract network in the deep neural network.

As shown in FIG. 9 , the apparatus further includes: a model training module 801 and an output module 808 . in:

The model training module 801 is used to obtain a sample data set; the sample data set includes a first set of real scene sample graphs; the sample data set is input into the deep neural network to be trained for iterative training, and the structural reconstruction loss is determined in each iteration value and bilateral filtering smoothing loss value, and reconstruct the loss value and bilateral filtering smoothing loss value according to the structure to determine the target loss value; in each round of iteration, adjust the network model parameters according to the target loss value until the iteration is stopped and the training is obtained The completed deep neural network; the network model parameters include the parameters of the first generator; the trained deep neural network includes the trained abstract network; wherein, the structural reconstruction loss value is used to indicate that the first generator is based on real scene samples The difference in structural features between the cartoon image generated by the graph and the sample graph of the real scene; the bilateral filtering smoothes the loss value, which is used to determine the adjacent pixels in the cartoon image generated by the first generator according to the similarity of pixel values and the similarity of spatial positions difference between.

In one embodiment, the abstract processing module 804 is further configured to perform stylization processing on the abstract cartoon image through the first generator to generate a style cartoon image with artistic style.

In one embodiment, the abstract network further includes a first discriminator; the network model parameters further include parameters of the first discriminator; the sample data set further includes a second set of abstract cartoon sample images; the model training module 801 is further configured to In the round of iterations, the structure reconstruction loss value, the bilateral filter smoothing loss value and the style enhancement loss value are determined, and the target loss value is determined according to the structure reconstruction loss value, the bilateral filter smoothing loss value and the style enhancement loss value; The loss value is determined by the first discriminator according to the first probability output by the cartoon image generated by the first generator; the first probability is used to represent the probability that the cartoon image generated by the first generator belongs to the style of the abstract cartoon sample image .

In one embodiment, the model training module 801 is further configured to obtain a third set of original cartoon images; the original cartoon images in the third set have the same artistic style; according to the pre-trained line tracing network, extract the Contour edge lines; according to the extracted contour edge lines and the original cartoon image, an abstract cartoon sample image is generated, and the second set is obtained; the abstract cartoon sample image is an abstract image after the contour edge lines are eliminated from the original cartoon image.

In one embodiment, the deep learning network is a two-stage neural network, and the abstract network is a first-stage abstract network in the two-stage neural network; the two-stage neural network further includes a second-stage line drawing network; the line generation module 806 also uses In the second stage, the style cartoon image generated in the first stage is input to the second generator in the second stage line drawing network in the trained two-stage neural network, so that the style cartoon image is analyzed by the second generator. The image is subjected to contour edge line generation processing to obtain a cartoon image of the real scene image; wherein, the second stage line drawing network is a deep neural network that generates contour edge lines in the second stage.

In one embodiment, the abstract cartoon sample image is an abstract image that eliminates contour edge lines from the original cartoon image; the sample data set further includes a third set of original cartoon images corresponding to the abstract cartoon sample image; the network model parameters Also includes parameters for the second generator. The model training module 801 is further configured to determine the structural reconstruction loss value, the bilateral filtering smoothing loss value, the style enhancement loss value and the edge line distribution loss value in each iteration, and according to the structural reconstruction loss value, the bilateral filtering smoothing loss value , the style enhancement loss value and the edge line distribution loss value to determine the target loss value. Among them, the edge line distribution loss value is used to represent the image with lines generated when the abstract cartoon sample image is used as the input of the second generator, and The difference between the original cartoon images corresponding to the abstract cartoon sample images; the network model parameters also include the parameters of the second generator.

In one embodiment, the second-stage line drawing network further includes a second discriminator; the network model parameters further include parameters of the second discriminator; the model training module 801 is further configured to determine the structural reconstruction loss value, bilateral filter smoothing loss value, style enhancement loss value, edge line assignment loss value and edge line enhancement loss value; according to the structure reconstruction loss value, bilateral filter smoothing loss value, style enhancement loss value, edge line assignment loss value and the The edge line enhances the loss value to determine the target loss value. Among them, the loss value of edge line enhancement is the image with lines generated when the second generator takes the image generated by the first generator as input, and after inputting it to the second discriminator, the second discriminator outputs the first image. The second probability is determined; the second probability is used to characterize the line intensity difference between the image with lines generated by the second generator and the original cartoon image.

In one embodiment, the iterative training includes structural reconstruction iterative training and network stacking iterative training; the structure reconstruction iterative training is used to train the initialization abstract network for image reconstruction processing; the network stacking iterative training refers to combining the initialization abstract network and the first Two-stage line delineation networks, layered together for iterative training.

In this embodiment, the model training module 801 is further configured to obtain a first set of real scene sample images, a second set of abstract cartoon sample images, and a third set of original cartoon images corresponding to the abstract cartoon sample images; The first set is input to the abstract network to be trained for structural reconstruction iterative training, the structural reconstruction loss value is determined in each round of structural reconstruction iterative training, and the parameters of the first generator are adjusted according to the structural reconstruction loss value until the initialization training stops. condition, to obtain the initialized abstract network; input the first set and the second set to the initialized abstract network, and input the second set and the third set to the second-stage line drawing network to be trained for network cascade iterative training, and at each Determine the structure reconstruction loss value, the bilateral filter smoothing loss value, the style enhancement loss value, and the edge line assignment loss value in the iterations; according to the structure reconstruction loss value, the bilateral filter smoothing loss value, the style enhancement loss value, and the edge line assignment Loss value, determine the comprehensive loss value, and according to the comprehensive loss value, adjust the parameters of the first generator and the first discriminator to initialize the abstract network, and adjust the second generator and second generator of the line drawing network in the second stage to be trained. The parameters of the discriminator until the iteration stops, and the trained two-stage neural network is obtained; the two-stage neural network includes the first-stage abstract network and the second-stage line drawing network; among them, the comprehensive loss value is the network layered iteration training each time. The target loss value determined in the round iteration.

For the specific limitation of the processing device for the cartoonization of the real scene image, please refer to the limitation of the processing method for the cartoonization of the real scene image above, which will not be repeated here. Each module in the above-mentioned real scene image cartoonization processing device may be implemented in whole or in part by software, hardware and combinations thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

In one embodiment, a computer device is provided, and the computer device may be a terminal, and its internal structure diagram may be as shown in FIG. 10 . The computer equipment includes a processor, memory, a communication interface, a display screen, and an input device connected by a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium, an internal memory. The nonvolatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the execution of the operating system and computer programs in the non-volatile storage medium. The communication interface of the computer device is used for wired or wireless communication with an external terminal, and the wireless communication can be realized by WIFI, operator network, NFC (Near Field Communication) or other technologies. When the computer program is executed by the processor, a processing method for cartoonizing a real scene image is realized. The display screen of the computer equipment may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment may be a touch layer covered on the display screen, or a button, a trackball or a touchpad set on the shell of the computer equipment , or an external keyboard, trackpad, or mouse.

In one embodiment, a computer device is provided, and the computer device may be a server, and its internal structure diagram may be as shown in FIG. 11 . The computer device includes a processor, memory, and a network interface connected by a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium, an internal memory. The nonvolatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the execution of the operating system and computer programs in the non-volatile storage medium. The computer device's database is used to store search data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a processing method for cartoonizing a real scene image is realized.

Those skilled in the art can understand that the structures shown in FIGS. 10 and 11 are only block diagrams of partial structures related to the solution of the present application, and do not constitute a limitation on the computer equipment to which the solution of the present application is applied. A device may include more or fewer components than shown in the figures, or combine certain components, or have a different arrangement of components.

In one embodiment, a computer device is also provided, including a memory and a processor, where a computer program is stored in the memory, and the processor implements the steps in the foregoing method embodiments when the processor executes the computer program.

In one embodiment, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, implements the steps in the foregoing method embodiments.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage In the medium, when the computer program is executed, it may include the processes of the above-mentioned method embodiments. Wherein, any reference to memory, storage, database or other media used in the various embodiments provided in this application may include at least one of non-volatile and volatile memory. The non-volatile memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, and the like. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, the RAM may be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM).

The technical features of the above embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features It is considered to be the range described in this specification.

The above-mentioned embodiments only represent several embodiments of the present application, and the descriptions thereof are specific and detailed, but should not be construed as a limitation on the scope of the invention patent. It should be pointed out that for those skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the scope of protection of the patent of the present application shall be subject to the appended claims.

Claims (15)

1. A method for processing cartoon of real scene image is characterized by comprising the following steps:

acquiring a real scene image;

based on the semantic information of the real scene image, carrying out image reconstruction processing and abstract smoothing processing on the real scene image to obtain an abstract cartoon image of the real scene image mapped on a cartoon domain; the abstract cartoon image has a significant structure in the real scene image and is missing a contour edge line in the real scene image;

stylizing the abstract cartoon image to generate a style cartoon image with artistic style;

and generating contour edge lines of the style cartoon image to obtain the cartoon image after the cartoon image of the real scene is cartoon-ized.

2. The method according to claim 1, wherein the real scene image is extracted from a sequence of video frames in sequence; the video frame sequence comprises at least two real scene images which are arranged in sequence;

the method further comprises the following steps:

when the sequence of video frames is a sequence of image frames in a video file, then

Generating cartoon video files according to cartoon images cartoon after cartoon of each real scene image in the video frame sequence,

or when the video file is played, the cartoon images corresponding to the real scene images in the video frame sequence are acquired in real time and output.

3. The method of claim 2, further comprising:

when the sequence of video frames is a sequence of image frames in a real-time video stream, then

And outputting cartoon video stream according to the cartoon image after cartoon of the real scene image in the video frame sequence.

4. The method according to any one of claims 1 to 3, wherein the performing image reconstruction processing and abstract smoothing processing on the real scene image based on semantic information of the real scene image to obtain an abstract cartoon image of the real scene image mapped on a cartoon domain comprises:

inputting the real scene image into a trained deep neural network to perform semantic extraction on the real scene image in a first stage;

based on the extracted semantic information, carrying out image reconstruction processing on the real scene image;

and carrying out bilateral filtering smoothing treatment on the reconstructed image content to generate an abstract cartoon image.

5. The method of claim 4, wherein the real scene image is input into a first generator of an abstract network in the deep neural network; the training step of the deep neural network comprises the following steps:

acquiring a sample data set; the sample data set comprises a first set of real scene sample graphs;

inputting the sample data set into a deep neural network to be trained for iterative training, determining a structural reconstruction loss value and a bilateral filtering smoothing loss value in each iteration, and determining a target loss value according to the structural reconstruction loss value and the bilateral filtering smoothing loss value;

in each iteration, adjusting network model parameters according to the target loss value until the iteration is stopped to obtain a trained deep neural network; the network model parameters include parameters of the first generator; the trained deep neural network comprises a trained abstract network;

the structure reconstruction loss value is used for representing the difference of the structural characteristics between the cartoon image generated by the first generator according to the real scene sample image and the real scene sample image;

and the bilateral filtering smoothing loss value is used for determining the difference between adjacent pixels in the cartoon image generated by the first generator according to the pixel value similarity and the spatial position similarity.

6. The method of claim 5, wherein said stylizing said abstract cartoon image to produce a stylized cartoon image having an artistic style comprises:

in the first stage, stylizing the abstract cartoon image through the first generator to generate a style cartoon image with artistic style.

7. The method of claim 6, wherein the abstract network further comprises a first arbiter; the network model parameters further comprise parameters of the first discriminator; the sample data set further comprises a second set of abstract cartoon sample diagrams;

determining a structural reconstruction loss value and a bilateral filtering smoothing loss value in each iteration, and determining a target loss value according to the structural reconstruction loss value and the bilateral filtering smoothing loss value comprises:

in each iteration, determining a structural reconstruction loss value, a bilateral filtering smoothing loss value and a grid enhancement loss value, and determining a target loss value according to the structural reconstruction loss value, the bilateral filtering smoothing loss value and the grid enhancement loss value;

wherein the style enhancement loss value is determined by the first discriminator based on a first probability of the cartoon image output generated by the first generator; the first probability is used for representing the probability that the cartoon image generated by the first generator belongs to the style of the abstract cartoon sample graph.

8. The method of claim 7, wherein the obtaining of the second set comprises:

acquiring a third set of original cartoon images; the original cartoon images in the third set have the same artistic style;

extracting contour edge lines of the original cartoon images according to a pre-trained line tracking network;

generating an abstract cartoon sample picture according to the extracted contour edge lines and the original cartoon image to obtain a second set; the abstract cartoon sample picture is an abstract picture obtained by eliminating contour edge lines from the original cartoon picture.

9. The method of claim 7, wherein the deep learning network is a two-stage neural network, and the abstraction network is a first-stage abstraction network of the two-stage neural network; the two-stage neural network also comprises a second-stage line drawing network; the generating of the contour edge line of the style cartoon image to obtain the cartoon image comprises:

in the second stage, inputting the style cartoon image generated in the first stage into a second generator in a second-stage line drawing network in the trained two-stage neural network, and performing contour edge line generation processing on the style cartoon image through the second generator to obtain a cartoon image after cartoon of the real scene image is cartoonized;

wherein the second-stage line tracing network is a deep neural network for generating contour edge lines in the second stage.

10. The method of claim 9, wherein the abstract cartoon sample map is an abstract image with contour edge lines removed from an original cartoon image; the sample data set further comprises a third set of the original cartoon images corresponding to the abstract cartoon sample map; the network model parameters further include parameters of the second generator;

in each iteration, determining a structure reconstruction loss value, a bilateral filtering smoothing loss value and a style enhancement loss value, and determining a target loss value according to the structure reconstruction loss value, the bilateral filtering smoothing loss value and the style enhancement loss value comprises:

determining a structure reconstruction loss value, a bilateral filtering smoothing loss value, a style enhancement loss value and an edge line distribution loss value in each iteration, and determining a target loss value according to the structure reconstruction loss value, the bilateral filtering smoothing loss value, the style enhancement loss value and the edge line distribution loss value;

and the edge line distribution loss value is used for representing the difference between the image with the line, which is generated when the abstract cartoon sample diagram is used as the input of the second generator, and the original cartoon image corresponding to the abstract cartoon sample diagram.

11. The method of claim 10, wherein the second stage line drawing network further comprises a second arbiter; the network model parameters further comprise parameters of the second discriminator;

in each iteration, determining a structure reconstruction loss value, a bilateral filtering smoothing loss value, a style enhancement loss value and an edge line distribution loss value, and determining a target loss value according to the structure reconstruction loss value, the bilateral filtering smoothing loss value, the style enhancement loss value and the edge line distribution loss value, including:

in each iteration, determining a structure reconstruction loss value, a bilateral filtering smoothing loss value, a style enhancement loss value, an edge line distribution loss value and an edge line enhancement loss value;

determining a target loss value according to the structure reconstruction loss value, the bilateral filtering smoothing loss value, the style enhancement loss value, the edge line distribution loss value and the edge line enhancement loss value;

the edge line enhancement loss value is determined by a second probability output by the second discriminator after the image with the line generated by the second generator with the image generated by the first generator as input is input to the second discriminator; and the second probability is used for representing the line intensity difference between the image with the lines generated by the second generator and the original cartoon image.

12. The method of claim 4, wherein the deep neural network training step comprises:

acquiring a first set of real scene sample images, a second set of abstract cartoon sample images and a third set of original cartoon images corresponding to the abstract cartoon sample images;

inputting the first set into an abstract network to be trained for structure reconstruction iterative training, determining a structure reconstruction loss value in each round of structure reconstruction iterative training, and adjusting parameters of the first generator according to the structure reconstruction loss value until an initialization training stop condition is reached to obtain an initialization abstract network;

inputting the first set and the second set into the initialized abstract network, inputting the second set and the third set into a second-stage line drawing network to be trained for network stacking iterative training, and determining a structure reconstruction loss value, a bilateral filtering smoothing loss value, a style enhancement loss value and an edge line distribution loss value in each iteration;

determining a comprehensive loss value according to the structure reconstruction loss value, the bilateral filtering smoothing loss value, the style enhancement loss value and the edge line distribution loss value;

adjusting parameters of the first generator and the first discriminator of the initialized abstract network and parameters of the second generator and the second discriminator of the second-stage line drawing network to be trained according to the comprehensive loss value until iteration stops to obtain a trained two-stage neural network; the two-stage neural network comprises a first-stage abstract network and a second-stage line drawing network.

13. A device for processing cartoon of real scene image, the device comprising:

the acquisition module is used for acquiring a real scene image;

the abstract processing module is used for carrying out image reconstruction processing and abstract smoothing processing on the real scene image based on the semantic information of the real scene image to obtain an abstract cartoon image of the real scene image mapped on a cartoon domain; the abstract cartoon image has a significant structure in the real scene image and is missing a contour edge line in the real scene image; stylizing the abstract cartoon image to generate a style cartoon image with artistic style;

and the line generating module is used for generating contour edge lines of the style cartoon image to obtain the cartoon image after the cartoon image of the real scene is cartoon-ized.

14. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor, when executing the computer program, implements the steps of the method of any of claims 1 to 12.

15. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 12.

Images