CN114897916A - Image processing method and device, nonvolatile readable storage medium and electronic equipment - Google Patents

Abstract

The application discloses an image processing method and device, a nonvolatile readable storage medium and electronic equipment. Wherein, the method comprises the following steps: acquiring first image data of a target object and second image data of a target scene; segmenting the first image data according to the type of the first image data to obtain a first target area corresponding to the target object in the first image data; and fusing the first target area and the second image data to generate a fused image. The method and the device solve the technical problem that the background environment cannot be considered due to the lack of the respective characteristics of the front camera and the rear camera.

Description

Image processing method and apparatus, non-volatile readable storage medium, electronic device

technical field

The present application relates to the technical field of image processing, and in particular, to an image processing method and apparatus, a non-volatile readable storage medium, and an electronic device.

Background technique

The camera is one of the most important sensors in a smart terminal. Using the front camera to take selfies and the rear camera to capture other people and landscapes, whether for storage or sharing, has become the most important part of the daily life of many people today. At present, the vast majority of cameras and applications in smart terminals are only based on one of the front camera or the rear camera to obtain image or video data and process them. Interesting portrait image or video algorithm special effects. And when taking a selfie, the photographer's attention will focus on himself, and cannot take into account the background environment.

For the above problems, no effective solution has been proposed yet.

SUMMARY OF THE INVENTION

Embodiments of the present application provide an image processing method and device, a non-volatile readable storage medium, and an electronic device, to at least solve the problem of inability to take into account the background environment due to the lack of effective use of the respective characteristics of the front camera and the rear camera. technical problem.

According to an aspect of the embodiments of the present application, an image processing method is provided, including: acquiring first image data of a target object and second image data of a target scene; The image data is segmented to obtain a first target area corresponding to the target object in the first image data; the first target area and the second image data are fused to generate a fused image.

Optionally, performing segmentation processing on the first image data according to the type of the first image data to obtain a first target area corresponding to the target object in the first image data, including: when the first image data is An image data is a video, and the first image data is segmented by using a pre-trained first neural network model to obtain a first target area corresponding to the target object; when the first image data is a picture, use The predetermined second neural network model performs segmentation processing on the first image data to obtain a first target area corresponding to the target object.

Optionally, before the category judgment is performed on the first image data, the method further includes: performing content recognition and judgment on the first image data; when the input image data is judged to belong to a scene supported by the segmentation process When it is determined that the first image data does not belong to the scene supported by the segmentation process, the first image data is determined to be a category; when the input image data is determined not to belong to a scene supported by the segmentation process, the first image data ends the process.

Optionally, the first neural network model is a lightweight model in which a backbone network and a decoding network are connected across layers as a network structure.

Optionally, the training method of the pre-trained first neural network model includes: acquiring a training data set, wherein the training data set includes: first sample image data and a first sample target area, wherein the The first sample target area is to obtain a target area mask map based on the first sample image data; the neural network model is trained based on the training data set, and the first neural network model is generated, wherein, in the training of the During the process of the first neural network model, consistency constraints are imposed on the first neural network model based on the inter-frame information.

Optionally, after using the pre-trained first neural network model to obtain the first target area corresponding to the target object, the method further includes: converting the first target area into a grayscale mask, and converting the first target area into a grayscale mask. The boundaries of the grayscale mask are smoothed.

Optionally, after obtaining the first target area corresponding to the target object by using the pre-trained first neural network model, the method further includes: obtaining a previous frame segmentation result of the first image data, and using the The first target area corresponding to the target is subjected to temporal smoothing filtering on the result of the previous frame segmentation.

Optionally, the second neural network model is a convolutional neural network model with atrous convolution adoption and attention mechanism added.

Optionally, after obtaining the first target area corresponding to the target object by using a pre-trained second neural network model, the method further includes: combining the first image data and using a pre-trained third neural network Optimizing the first target area is performed to obtain a processed first target area.

Optionally, the training method of the pre-trained third neural network model includes: acquiring an image of a solid-color background; performing matting processing and pre-labeling on the image of the solid-color background to obtain a label mask image; The third neural network model is obtained by training the background image and the label mask image as sample data.

Optionally, before performing fusion processing on the first target area and the second image data to generate a fused image, the method further includes: inputting the first target area into a post-processing module for post-processing.

Optionally, performing fusion processing on the first target area and the second image to generate a fused image includes: evaluating the environmental information of the second image data, and correcting the first target area according to the environmental information. target area, obtain the corrected first target area; determine the second target area corresponding to the first target area in the second image data; replace the second target area with the corrected first target area target area.

According to another aspect of the embodiments of the present application, an image processing apparatus is further provided, including: an acquisition module configured to acquire first image data of a target object and second image data of a target scene, wherein the first image data is acquired The device for collecting image data and the second image data is located on the same terminal device; the segmentation module is used to perform segmentation processing on the first image data to obtain the first image data corresponding to the target object. a first target area; a fusion module, configured to perform fusion processing on the first target area and the second image data to generate a fused image.

According to yet another aspect of the embodiments of the present application, a non-volatile storage medium is also provided, the non-volatile storage medium includes a stored program, wherein when the program runs, the device where the non-volatile storage medium is located is controlled to execute the above image processing method.

According to another aspect of the embodiments of the present application, an electronic device is also provided, including a memory and a processor; the processor is used for running a program, wherein the above-mentioned image processing method is executed when the program is running.

In the embodiment of the present application, the first image data of the target object and the second image data of the target scene are acquired; the first image data is segmented according to the type of the first image data, and the first image data is obtained. The first target area corresponding to the target object in the image data; the method of merging the first target area and the second image data to generate a fused image is performed by the acquisition device on the same terminal device. Collect the first image and the second image and fuse the first image and the second image to achieve the purpose of making full use of the front and rear cameras, thereby realizing the technical effect of taking into account the background environment, and solving the problem of lack of effective utilization. Due to the respective characteristics of the front camera and the rear camera, it is impossible to take into account the technical problems of the background environment.

Description of drawings

The drawings described herein are used to provide further understanding of the present application and constitute a part of the present application. The schematic embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation of the present application. In the attached image:

1 is a schematic diagram of an optional image processing method according to an embodiment of the present application;

2 is a schematic diagram of another optional image processing method according to an embodiment of the present application;

3 is a schematic diagram of an optional image segmentation processing flow according to an embodiment of the present application;

FIG. 4 is a schematic diagram of an optional image processing apparatus according to an embodiment of the present application.

Detailed ways

In order to make those skilled in the art better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only The embodiments are part of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the scope of protection of the present application.

It should be noted that the terms "first", "second", etc. in the description and claims of the present application and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence. It is to be understood that data so used may be interchanged under appropriate circumstances so that the embodiments of the application described herein can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having" and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those expressly listed Rather, those steps or units may include other steps or units not expressly listed or inherent to these processes, methods, products or devices.

According to an embodiment of the present application, an embodiment of a method for processing an image is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings may be executed in a computer system such as a set of computer-executable instructions, and, Although a logical order is shown in the flowcharts, in some cases steps shown or described may be performed in an order different from that herein.

FIG. 1 is an image processing method according to an embodiment of the present application. As shown in FIG. 1 , the method includes the following steps:

Step S102, obtaining the first image data of the target object and the second image data of the target scene;

Step S104, performing segmentation processing on the first image data according to the type of the first image data to obtain a first target area corresponding to the target object in the first image data;

Step S106, performing fusion processing on the first target area and the second image data to generate a fused image.

Through the above steps, the acquisition device located on the same terminal device can collect the first image data and the second image data respectively, and fuse the target object contained in the first image data with the second image data, so as to make full use of the front-end image data. In this way, the technical effect of taking into account the background environment is achieved, thereby solving the technical problem that the background environment cannot be taken into account due to the lack of effective use of the respective characteristics of the front camera and the rear camera.

It should be noted that the above-mentioned image data collection devices are the front camera and the rear camera of the same terminal (for example, a mobile phone and a notebook), and the collected image data that includes the target object is the first image data, so the first image data The image data acquisition device is not fixed to a front camera, but can also be a rear camera. Similarly, the second image data acquisition device is a front camera or a rear camera; the first image data and the second image data can be pictures or It can be a video; the target object can be a portrait or other objects, such as animals and objects; the target scene can be the scene where the target object is located or any virtual scene.

Specifically, taking the use of the front camera of the mobile phone to shoot the portrait image and the rear camera to shoot the background video as an example, the steps of image processing can be as shown in Figure 2. The collected portrait image and background video are fused after different processing procedures. The merged video is obtained. On the one hand, the foreground and background information can be collected at the same time, and more scene information of the shooting location can be preserved and restored; on the other hand, the advantages of the front and rear cameras of the mobile phone are fully utilized. , which not only facilitates observation of the imaging situation of the self-portrait portrait, but also facilitates observation of the information of the entire background scene for better framing.

In some embodiments of the present application, the first image data is segmented according to the type of the first image data to obtain a first target area corresponding to the target object in the first image data, including : when the first image data is a video, use the pre-trained first neural network model to segment the first image data to obtain the first target area corresponding to the target object; when the first image The data is a picture, and the first image data is segmented by using a predetermined second neural network model to obtain a first target area corresponding to the target object.

It should be noted that, since this application does not limit the types of data collected by the terminal, the collected first image data and second image data may be pictures or videos. In actual segmentation, the processing targets for different types of data are Different from the method, before the segmentation processing of the image data, the category judgment is performed on the above-mentioned first image data. Further, for the segmentation processing of video data, it is necessary to not only ensure the accuracy but also maintain real-time processing. Exemplarily, the first neural network model may be a lightweight neural network model, which is used for segmentation processing of video images; For data segmentation processing, since images have higher requirements on details, exemplary, the second neural network model can be a convolutional neural network model.

In some embodiments of the present application, before the category judgment is performed on the first image data, the method further includes: performing content recognition and judgment on the first image data; when the input image data is judged to belong to segmentation When processing a supported scene, the first image data performs category judgment; when it is judged that the input image data does not belong to a scene supported by the segmentation process, the first image data ends the processing.

Specifically, taking the image data of the mobile device as the input as an example, the extraction of the target area in the picture is realized by the method of computer vision. The processing flow is shown in Figure 3: input image data and then preprocess the image, Afterwards, it is judged whether it belongs to the scene supported by the segmentation engine, and if it is supported, it will be processed, and if it is not supported, it will not be processed. First, the scene supported by the segmentation engine requires the image to contain the target object, where the type of the target object is preset by the user, and the type of the target object contained in the image is obtained through the detection and recognition algorithm; secondly, the distance between the target object and the acquisition device needs to meet the predetermined Setting conditions, if the distance between the target object and the acquisition device exceeds the preset range, too close will result in incomplete acquisition of the target object. For example, only part of the facial features cannot form a group photo. If the distance is too far, the details of the acquired target object will be seriously lost and seriously affected. The subsequent fusion effect, so the acquisition device cannot be activated.

It should be noted that the preprocessing of the image data is to take the image data of the mobile device as the input, convert it into the corresponding data format according to the subsequent segmentation network requirements, and adjust the size, color, angle, etc. of the image to obtain the available data. Image data; identifying whether the preprocessed image belongs to the scene supported by the segmentation engine is composed of a scene discrimination convolutional neural network pre-trained based on massive labeled supervision data, which can quickly and accurately identify the content of the input image, Only when the input image is determined to belong to the scene supported by the segmentation engine, the subsequent processing flow is entered. The recognition network is a typical classification network composed of a stack of convolutional layers, activation layers, and pooling layers. Considering the performance requirements of mobile devices, the number and width of the network are strictly limited and optimized to ensure millisecond-level operation on the device. .

In some embodiments of the present application, the first neural network model is a lightweight model using a cross-layer connection between a backbone network and a decoding network as a network structure. The training method of the pre-trained first neural network model includes: acquiring a training data set, wherein the training data set includes: first sample image data and a first sample target area, wherein the first sample This target area is to obtain a target area mask map based on the first sample image data; train a neural network model based on the training data set to generate the first neural network model, wherein, when training the first neural network In the process of modeling, the first neural network model is subject to consistency constraints based on the inter-frame information.

Specifically, for the first neural network model, the training data set is first established by collecting a large amount of first sample image data, wherein the first sample image data needs to include a group photo of the object and the background at the same time. The networks are independent of each other, and the sample image data does not restrict the objects and scenes to be the target object and target background for recognition, but the sample image data needs to cover the actual use scene category. In addition, the present application does not limit the method of generating the first sample image data, which can be collected in actual scenes or synthesized in a later stage. Further, after acquiring the first sample image data, the target area of the first sample is determined by manual or automatic identification, that is, the target area mask map is obtained by annotating the first sample image data as the supervision information. Considering the real-time requirements of video applications, the network part adopts a lightweight model, which consists of a convolution layer, an activation layer, a pooling layer, and a deconvolution layer. Targeted optimization has been made in terms of the number of network layers, convolution types, and downsampling positions. The network structure adopts the cross-layer connection of the backbone network and the decoding network to improve the segmentation accuracy. At the same time, in the network deployment, with the help of the instruction optimization of the running hardware platform and the optimization of special equipment, the computing performance of real-time segmentation can be achieved. In the training process, in order to improve the stability of the video timing results, the inter-frame information consistency constraint is added to the training process.

In some embodiments of the present application, after obtaining the first target area corresponding to the target object by using the pre-trained first neural network model, the method further includes: converting the first target area into a grayscale mask code image; and smooth the boundary of the grayscale mask image.

Specifically, after the video obtains the corresponding first target area through the pre-trained first neural network model, the first target area is converted into a grayscale mask image with a background value of 0. Secondly, the image processing algorithm removes small isolated areas and smoothes the mask boundary. The introduction of the mask is not only conducive to shielding the noise outside the target area, but also can fully extract and utilize the area of interest, thereby optimizing the segmentation results.

In some embodiments of the present application, after obtaining the first target area corresponding to the target object by using the pre-trained first neural network model, the method further includes: obtaining a previous frame segmentation result of the first image data , and use the previous frame segmentation result to perform temporal smoothing filtering on the first target area corresponding to the target. Specifically, the temporal smoothing filtering is performed by using the segmentation result of the previous frame, which increases the inter-frame stability and ensures the continuity of the output video result.

In some embodiments of the present application, the second neural network model is a convolutional neural network model with atrous convolution and an attention mechanism added.

Specifically, for the second neural network model, the network model is composed of a deep convolutional network with more parameters and a more complex structure. Taking into account the higher requirements for details in the photo mode, more refined standards are adopted when the training data is manually annotated, which improves the quality of the supervision data. At the same time, the requirements for computing power performance in the photo mode are reduced, so structures such as hole convolution and attention mechanism are appropriately added to the network part to improve the network analysis ability. The depth, width, and feature map size of the network have also been appropriately relaxed to achieve more accurate segmentation requirements. In the deployment stage, the optimization of instructions on the operating platform and the optimization of special equipment are also used to improve the running speed of the model.

It should be noted that the first target area may be the area where the target object is located in the first image data.

In some embodiments of the present application, after obtaining the first target area corresponding to the target object by using the pre-trained second neural network model, the method further includes: combining the first image data and using the pre-trained The third neural network of the device performs optimization processing on the first target area to obtain the processed first target area.

Wherein, the training method of the pre-trained third neural network model includes: acquiring an image of a solid-color background; performing matting processing and pre-labeling on the image of the solid-color background to obtain a label mask image; The image and the label mask image are sample data to train to obtain the third neural network model.

It should be noted that, as shown in FIG. 3 , the third neural network may be a Matting network (a neural network used for refined image segmentation).

Specifically, the input in the third neural network is the output of the second neural network model and the original first image data. Due to the limitations of segmentation network resolution and downsampling, the region map obtained by the second neural network cannot obtain fine segmentation results in areas such as object edges and hair. Taking the Matting network as an example to illustrate the above steps, on the basis of the first target area output by the second neural network, the Matting network obtains the trimap map (triple map) according to the confidence, and at the same time adds an attention mechanism to the network, which can make the network Focus more on edges and improve edge accuracy. The output of the network is a mask map, which is the processed first target area map, which is a regression of the opacity of each pixel position.

Since the training of the third neural network requires more fine-grained transparency as the supervision data, this application adopts three steps of fixed-point collection, automatic pre-labeling, and manual correction to obtain the training data. Specifically, fixed-point collection refers to building a solid-color background collection environment with adjustable ambient light to obtain relatively natural solid-color background data. The automatic pre-labeling uses the image matting algorithm and the trained third neural network to process the solid color background data to obtain the initial mask map. Finally, the error area in the initial mask map result is fine-tuned by manual correction, and the final labeling result is obtained. During the implementation process, the third neural network of the automatic pre-labeling part can iterate with the data update, so that the pre-labeling effect is continuously improved, and the manual labeling cost is further gradually reduced.

In some embodiments of the present application, before performing fusion processing on the first target area and the second image data to generate a fused image, the method further includes: inputting the first target area into post-processing module for post-processing.

It should be noted that when the terminal device is movable, such as a mobile phone, it is often shot by a handheld device, and the captured video often contains a certain amount of jitter. The video anti-shake module can be used to reduce or eliminate it through hardware and software technology. Shaking in the video, making the captured video more stable. In this application, if the target image type includes video, an anti-shake module will be added to perform anti-shake processing.

In some optional ways, due to the limitation of the computing power and memory of the mobile phone, the neural network-based semantic segmentation algorithm can only process images with smaller resolutions, generally no larger than 512x512 pixels, which is relatively high-definition (720P) images compared to the original images. ), Ultra HD (1080P), or even 4K, which tend to be much smaller. The general processing method is to first downsample the image to the neural network resolution, and then upsample to the original resolution after obtaining the result. The result after direct upsampling and the original resolution are often misaligned on the boundary, and due to the scaling relationship, the details of the small image will be lost. After upsampling, the details that cannot be displayed on the small image will be lost. The obtained segmentation results are also inaccurate. Based on this, the present application adopts a post-processing module, and based on the segmentation results of the first neural network, the second neural network or the third neural network, the boundary alignment with the original resolution image calculated by the edge filter algorithm is more accurate, and the details are more accurate. Rich portrait foreground area.

In some embodiments of the present application, performing fusion processing on the first target area and the second image data to generate a fused image includes: estimating the environment information of the second image data, according to the environment information to correct the first target area to obtain the corrected first target area; determine the second target area corresponding to the first target area in the second image data; replace the second target area with the The corrected first target area.

Fusion is based on the characteristics of the front and the background to make the result as natural as possible, including retaining the naturalness of the foreground skin color, and adjusting the color tone of the foreground can more naturally adapt to the color tone of the background, so that the fused image visually forms a complete whole. And reduce the cutout marks. The present application does not limit the types of objects to be fused, and the fusion of image foreground and image background, image foreground and video background, video foreground and video background, and video foreground and image background can be implemented. Exemplarily, taking the foreground area of the portrait as the first target area and the background video result as the second image, the resultant video of the output portrait and the background video is fused.

Specifically, due to the difference in the position and image quality of the foreground and background target objects, direct replacement in the fusion module will lead to inconsistent final results. Therefore, this application will first evaluate the environmental information of the background, including the direction, intensity, color and other information of the illumination, and then correct the foreground accordingly to ensure the uniform image quality of the front and background images. Secondly, for the placement position, the position where the foreground area is placed in the background area is generally placed in the center. In addition, any one or more boundaries of the upper, lower, left, and right of the foreground image touch the image boundary, and the fusion result will also retain this feature. In practical applications, the collected background image to be fused is not a pure background, and there may be occlusions, such as persons and objects that are not target objects in the area. For the occlusion of the images in the above-mentioned areas, this application also includes Perform target recognition on the background image, and prompt the user when the recognition result contains an occluded target.

In addition, if the object to be fused includes video, the amount of calculation involved in the fusion calculation is huge. In order to not only ensure accuracy but also maintain real-time processing, simplify the processing of the corresponding object, including optimizing the overall process, and only perform lightweight network segmentation. , instead of using a fine-level network, and simplifying each module accordingly, such as reducing or even completely removing post-processing, and further compressing the model for the lightweight network used.

The embodiment of the present application also provides an image processing apparatus, as shown in FIG. 4 , including: an acquisition module 40 for acquiring first image data of a target object and second image data of a target scene; a segmentation module 42 for acquiring The first image data is segmented according to the type of the first image data to obtain a first target area corresponding to the target object in the first image data; the fusion module 44 is to divide the first target The region is fused with the second image data to generate a fused image.

The segmentation module 42 includes: a first processing sub-module; the judging sub-module is used to perform category judgment on the first image data; when the first image data is a video, the pre-trained first neural network model is used to classify the first image data. Perform segmentation processing on image data to obtain a first target area corresponding to the target object; when the first image data is a picture, use a predetermined second neural network model to perform segmentation processing on the first image data, A first target area corresponding to the target object is obtained.

The judgment sub-module includes: a judgment unit, a first training unit, and a transformation unit;

The judging unit is used to identify and judge the content of the first image data. When the inputted image data is judged to belong to the scene supported by the segmentation process, the first image data is judged by category; When it is determined that it does not belong to a scene supported by the segmentation process, the first image data ends the process.

The first training unit is used to obtain a training data set, wherein the training data set includes: first sample image data and a first sample target area, wherein the first sample target area is based on the first sample target area. Obtain a target area mask map from sample image data; train a neural network model based on the training data set, and generate the first neural network model, wherein, in the process of training the first neural network model, based on the inter-frame information Consistency constraints are imposed on the first neural network model.

The conversion unit is used for converting the first target area into a grayscale mask image; and smoothing the boundary of the grayscale mask image.

The first processing sub-module is configured to combine the first image data, and use a pre-trained third neural network to perform optimization processing on the first target area to obtain a processed first target area.

The first processing submodule includes: a second training unit;

The second training unit is used to obtain an image of a solid-color background; performing matting processing and pre-labeling on the image of the solid-color background to obtain a label mask image; taking the image of the solid-color background as sample data, the label mask image The third neural network model is obtained by training as the supervision data.

The fusion module 44 includes: a second processing sub-module and a generating sub-module; the second processing sub-module is configured to input the first target region into a post-processing module for post-processing.

The generating submodule is used to evaluate the environmental information of the second image data, correct the first target area according to the environmental information, and obtain the corrected first target area;

determining a second target area corresponding to the first target area in the second image data;

The second target area is replaced with the corrected first target area.

According to yet another aspect of the embodiments of the present application, a non-volatile storage medium is also provided, the non-volatile storage medium includes a stored program, wherein when the program runs, the device where the non-volatile storage medium is located is controlled to execute the above image processing method.

The above-mentioned non-volatile storage medium is used for storing a program that performs the following functions: acquiring a first image of a target object and a second image of a target scene; performing segmentation processing on the first image data according to the type of the first image data , obtain the first target area corresponding to the target object in the first image; perform fusion processing on the first target area and the second image to generate a fused image.

According to another aspect of the embodiments of the present application, an electronic device is also provided, including a memory and a processor; the processor is used for running a program, wherein the above-mentioned image processing method is executed when the program is running.

The above-mentioned processor is used to run a program that performs the following functions: acquiring a first image of a target object and a second image of a target scene; performing segmentation processing on the first image data according to the type of the first image data to obtain a first image the first target area corresponding to the target object in the middle; the first target area and the second image are fused to generate a fused image.

The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

In the above-mentioned embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are only illustrative, for example, the division of units may be a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components may be combined or integrated into Another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of units or modules, and may be in electrical or other forms.

Units described as separate components may or may not be physically separated, and components shown as units may or may not be physical units, that is, may be located in one place, or may be distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.

The integrated unit, if implemented as a software functional unit and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of software products in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, and the computer software products are stored in a storage medium , which includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods in the various embodiments of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes .

The above are only the preferred embodiments of the present application. It should be pointed out that for those skilled in the art, some improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as The protection scope of this application.

Claims (15)

1. A method of processing an image, comprising:

acquiring first image data of a target object and second image data of a target scene;

segmenting the first image data according to the type of the first image data to obtain a first target area corresponding to the target object in the first image data;

and fusing the first target area and the second image data to generate a fused image.

2. The method according to claim 1, wherein performing segmentation processing on the first image data according to the type of the first image data to obtain a first target region corresponding to the target object in the first image data comprises:

when the first image data is a video, segmenting the first image data by using a pre-trained first neural network model to obtain a first target area corresponding to the target object;

when the first image data is a picture, the first image data is segmented by using a predetermined second neural network model to obtain a first target area corresponding to the target object.

3. The method of claim 1, wherein prior to performing the classification determination on the first image data, the method further comprises:

performing content identification and judgment on the first image data;

performing a category determination on the first image data when the input first image data is determined to belong to a scene supported by a segmentation process;

when the input first image data is determined not to belong to a scene supported by segmentation processing, the first image data ends processing.

4. The method of claim 2, wherein the first neural network model is a lightweight model that employs a cross-layer connection of a backbone network and a decoding network as a network structure.

5. The method of claim 2, wherein the method of training the pre-trained first neural network model comprises:

obtaining a training data set, wherein the training data set comprises: the image processing method comprises the steps of obtaining first sample image data and a first sample target area, wherein the first sample target area is a target area mask map obtained based on the first sample image data;

training a neural network model based on the training data set to generate the first neural network model, wherein in the process of training the first neural network model, consistency constraint is carried out on the first neural network model based on interframe information.

6. The method of claim 2, wherein after obtaining the first target region corresponding to the target object by using the pre-trained first neural network model, the method further comprises:

and converting the first target area into a gray mask image, and smoothing the boundary of the gray mask image.

7. The method of claim 2, wherein after obtaining the first target region corresponding to the target object by using the pre-trained first neural network model, the method further comprises:

and acquiring a previous frame segmentation result of the first image data, and performing time domain smoothing filtering on the first target region corresponding to the target by using the previous frame segmentation result.

8. The method of claim 2, wherein the second neural network model is a convolutional neural network model that incorporates a hole convolution and attention mechanism.

9. The method of claim 2, wherein after obtaining the first target region corresponding to the target object using a pre-trained second neural network model, the method further comprises:

and combining the first image data and utilizing a pre-trained third neural network to perform optimization processing on the first target area to obtain a processed first target area.

10. The method of claim 9, wherein the method of training the pre-trained third neural network model comprises:

acquiring an image of a solid background;

performing matting and pre-labeling on the image of the solid background to obtain a label mask image;

and training by taking the image of the solid background and the label mask image as sample data to obtain the third neural network model.

11. The method according to claim 1 or 9, wherein before performing the fusion process on the first target region and the second image data to generate a fused image, the method further comprises:

and inputting the first target area into a post-processing module for post-processing.

12. The method of claim 1, wherein fusing the first target region with the second image data to generate a fused image comprises:

evaluating environmental information of the second image data, correcting the first target area according to the environmental information, and obtaining a corrected first target area;

determining a second target area corresponding to the first target area in the second image data;

replacing the second target area with the corrected first target area.

13. An image processing apparatus characterized by comprising:

the system comprises an acquisition module, a processing module and a display module, wherein the acquisition module is used for acquiring first image data of a target object and second image data of a target scene;

the segmentation module is used for segmenting the first image data according to the type of the first image data to obtain a first target area corresponding to the target object in the first image data;

and the fusion module is used for fusing the first target area and the second image data to generate a fused image.

14. A non-volatile storage medium, comprising a stored program, wherein when the program runs, a device in which the non-volatile storage medium is located is controlled to execute the image processing method according to any one of claims 1 to 12.

15. An electronic device comprising a memory and a processor; the processor is configured to execute a program, wherein the program executes a method for processing an image according to any one of claims 1 to 12.

Images