WO2021083020A1

WO2021083020A1 - Image fusion method and apparatus, and storage medium and terminal device

Info

Publication number: WO2021083020A1
Application number: PCT/CN2020/122679
Authority: WO
Inventors: 陈曦
Original assignee: RealMe重庆移动通信有限公司
Priority date: 2019-11-01
Filing date: 2020-10-22
Publication date: 2021-05-06
Also published as: CN112767290B; CN112767290A

Abstract

An image fusion method and apparatus, and a storage medium and a terminal device, relating to the technical field of image processing. The method is applied to a terminal device having an image sensor and comprises: acquiring a first image and a second image collected by the image sensor, wherein the first image has first resolution, the second image has second resolution, and the first resolution is higher than the second resolution (S110); extracting an edge feature image from the first image (S120); performing super-resolution reconstruction on the second image, and obtaining a third image having the same resolution as the first resolution (S130); and fusing the edge feature image and the third image to obtain a final image (S140). Therefore, the problem of much noise of a high-pixel image is solved, and the quality of image photographing is improved.

Description

Image fusion method, image fusion device, storage medium and terminal equipment

This application claims the priority of the Chinese patent application filed on November 1, 2019, with the application number 201911057665.5, titled "Image fusion method, image fusion device, storage medium and terminal equipment", and the entire content of the Chinese patent application Incorporated herein by reference.

Technical field

The present disclosure relates to the field of image processing technology, and in particular to an image fusion method, an image fusion device, a computer-readable storage medium, and terminal equipment.

Background technique

Image noise (or noise) refers to the brightness or color information that does not exist in the object itself, but appears in the image. It is generated by the image sensor or signal transmission circuit. At present, improving the pixels of image sensors is a common development direction in the industry. For example, image sensors with millions or even tens of millions of pixels are usually used on mobile phones, which can support the shooting of ultra-high-definition photos.

Summary of the invention

The present disclosure provides an image fusion method, an image fusion device, a computer-readable storage medium, and a terminal device, so as to improve at least to a certain extent the problem of high noise in existing high-pixel pictures.

According to a first aspect of the present disclosure, an image fusion method is provided, which is applied to a terminal device equipped with an image sensor, and the method includes: acquiring a first image and a second image collected by the image sensor, the first image Is the first resolution, the second image is the second resolution, the first resolution is higher than the second resolution; the edge feature image is extracted from the first image; the second image is processed Super-resolution reconstruction to obtain a third image, where the first image is the first resolution; and the edge feature image and the third image are fused to obtain a final image.

According to a second aspect of the present disclosure, there is provided an image fusion device configured in a terminal device equipped with an image sensor, the device including: an image acquisition module for acquiring a first image and a second image collected by the image sensor , The first image is a first resolution, the second image is a second resolution, and the first resolution is higher than the second resolution; an edge extraction module is used to extract the Extracting edge feature images; an image reconstruction module for super-resolution reconstruction of the second image to obtain a third image, the third image being the first resolution; a fusion processing module for converting the The edge feature image and the third image are fused to obtain the final image.

According to a third aspect of the present disclosure, there is provided a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned image fusion method and possible implementation manners thereof are realized.

According to a fourth aspect of the present disclosure, there is provided a terminal device including: a processor; a memory for storing executable instructions of the processor; and an image sensor; wherein the processor is configured to execute the executable Executing instructions to execute the above-mentioned image fusion method and its possible implementations.

The technical solution of the present disclosure has the following beneficial effects:

According to the above-mentioned image fusion method, image fusion device, storage medium and terminal equipment, a relatively high-resolution first image and a relatively low-resolution second image are collected, edge feature images are extracted from the first image, and the second image is super-resolved The rate is reconstructed into a third image, so that the resolution of the third image and the first image are the same, and finally the edge feature image and the third image are merged to obtain the final image. Among them, the resolution of the first image is higher, but there may be more noise. Through edge feature extraction, the original detail information in the image is preserved; the resolution of the second image is lower, but the noise is less. After reconstruction into the third image, the third image has less noise, but there may be distortion in details; after fusing the two parts of the image, it is equivalent to incorporating the original image detail information into the low-noise image, and the first image can be realized. The advantages of the first image and the second image are complementary to solve the problem of high-pixel image noise and improve the quality of image shooting.

Description of the drawings

Fig. 1 shows a flowchart of an image fusion method in this exemplary embodiment;

FIG. 2 shows a schematic diagram of a color filter array in this exemplary embodiment;

Fig. 3 shows a flowchart of a method for acquiring a first image and a second image in this exemplary embodiment;

Fig. 4 shows a schematic diagram of acquiring a first image in this exemplary embodiment;

Fig. 5 shows a schematic diagram of acquiring a second image in this exemplary embodiment;

Fig. 6 shows a flowchart of a method for extracting edge feature images in this exemplary embodiment;

FIG. 7 shows a schematic diagram of obtaining gradient difference images in this exemplary embodiment;

FIG. 8 shows a flowchart of a method for constructing a differential constraint item in this exemplary embodiment;

Fig. 9 shows a flowchart of a method for generating a third image in this exemplary embodiment;

Fig. 10 shows a flowchart of a method for generating a final image in this exemplary embodiment;

FIG. 11 shows a schematic diagram of an image fusion process in this exemplary embodiment;

FIG. 12 shows a schematic structural diagram of an image fusion device in this exemplary embodiment;

FIG. 13 shows a schematic structural diagram of another image fusion device in this exemplary embodiment;

FIG. 14 shows a schematic structural diagram of a terminal device in this exemplary embodiment.

Detailed ways

Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms, and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concept of the example embodiments will be fully conveyed To those skilled in the art. The described features, structures or characteristics can be combined in one or more embodiments in any suitable way. In the following description, many specific details are provided to give a sufficient understanding of the embodiments of the present disclosure. However, those skilled in the art will realize that the technical solutions of the present disclosure can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. can be used. In other cases, the well-known technical solutions are not shown or described in detail to avoid overwhelming the crowd and obscure all aspects of the present disclosure.

In addition, the drawings are only schematic illustrations of the present disclosure, and are not necessarily drawn to scale. The same reference numerals in the figures denote the same or similar parts, and thus their repeated description will be omitted. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in the form of software, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and/or processor devices and/or microcontroller devices.

With the increase of pixels, the photosensitive area of a single pixel on the image sensor decreases, and the requirements for illumination are higher when taking pictures. For example, in a low-light environment, the photosensitive element on the image sensor is more susceptible to crosstalk, resulting in insufficient signal-to-noise ratio of the input signal, and the final output image has too much noise and poor quality.

In the related art, the resolution of photographing is usually lowered actively when the light is insufficient, so as to increase the light-sensitivity and reduce the noise. However, this is at the cost of loss of resolution and cannot take advantage of the high-resolution image sensor itself.

In view of one or more of the above-mentioned problems, exemplary embodiments of the present disclosure provide an image fusion method, which can be applied to terminal devices such as mobile phones, tablet computers, and digital cameras. The terminal device is equipped with an image sensor, which can be used to collect images.

Fig. 1 shows a process of this exemplary embodiment, which may include the following steps S110 to S140:

Step S110, acquiring the first image and the second image collected by the image sensor;

Step S120, extract an edge feature image from the first image;

Step S130: Perform super-resolution reconstruction on the second image to obtain a third image;

In step S140, the edge feature image and the third image are merged to obtain a final image.

Each step in Figure 1 will be described in detail below.

In step S110, the first image and the second image collected by the image sensor are acquired.

Among them, the objects captured by the first image and the second image are the same, that is, the image content is the same, the difference lies in the image resolution, the first image is the first resolution, the second image is the second resolution, and the first resolution Higher than the second resolution, that is, the first image has more pixels than the second image.

In an embodiment, the above-mentioned image sensor may be a Quad Bayer (Quad Bayer) image sensor, and the Quad Bayer image sensor refers to an image sensor using a Quad Bayer color filter array. Refer to Figure 2, the left figure shows the standard Bayer color filter array, the unit array of the filter is GRBG (or BGGR, GBRG, RGGB), and most image sensors use the standard Bayer color filter array; Figure 2 The right figure shows a four-Bayer color filter array. The four adjacent cells in the unit array of the filter are of the same color. At present, some high-pixel image sensors use a four-Bayer color filter array. Based on this, referring to FIG. 3, step S110 can be specifically implemented through the following steps S310 to S330:

In step S310, the original Bayer image based on the four Bayer color filter array is collected by the four Bayer image sensor.

Among them, the Bayer image refers to an image in RAW format, which is image data after the image sensor converts the collected light signal into a digital signal. In a Bayer image, each pixel has only one color in RGB. In this exemplary embodiment, after the four Bayer image sensor is used to collect the image, the original image data obtained is the original Bayer image. The color arrangement of the pixels in the image is as shown in the right figure in Figure 2, and the adjacent four pixels are the same. colour.

Step S320: Perform demosaic processing and demosaic processing on the original Bayer image to obtain a first image.

Among them, the first image is the first resolution; Remosaic refers to the fusion of the original Bayer image based on the four Bayer color filter array into the Bayer image based on the standard Bayer color filter array; Demosaic refers to Fusion of Bayer images into complete RGB images. As shown in Figure 4, the original Bayer image P can be demosaiced to obtain the Bayer image Q1 based on the standard Bayer color filter array; then the Bayer image Q1 based on the standard Bayer color filter array can be demosaiced to obtain the RGB format The first image IMG1. Demosaicing and demosaicing can be implemented by different interpolation algorithms, and can also be implemented by other related algorithms such as neural networks, which are not limited in the present disclosure. The terminal device is usually equipped with an ISP (Image Signal Processing, image signal processing) unit that is matched with the image sensor to perform the above-mentioned demosaic and demosaic processing process. Each pixel of the first image IMG1 has pixel values of three RGB channels, denoted by C. In addition, the process of demosaicing and demosaicing can also be combined into one interpolation process, that is, based on the pixel data in the original Bayer image, each pixel is directly interpolated to obtain the pixel value of the missing color channel. For example, you can Use linear interpolation, mean interpolation and other algorithms to achieve, so as to obtain the first image.

In step S330, four adjacent pixels of the same color in the original Bayer image are merged into one pixel, and the merged Bayer image is demosaiced to obtain a second image.

The second image is the second resolution. As shown in Figure 5, the original Bayer image P is first processed by pixel "four-in-one", that is, the pixels of the same color in 2*2 units are combined into one pixel. The combined Bayer image Q2 is also based on the standard Bayer image. The arrangement of the color filter array, compared with Q1 in Figure 4, the pixels of Q2 are reduced to 1/4, and the area of each pixel is increased by 4 times, so that the amount of light entering each pixel increases; and then Q2 Perform demosaicing to obtain the second image IMG2 in RGB format. It can be seen that the first resolution is four times the second resolution.

In another implementation manner, the terminal device may be configured with two image sensors with different pixels. For example, many mobile phones are currently equipped with dual cameras. Among them, the first image sensor with higher pixels is used to capture the first image, and the second image sensor with lower pixels is used to capture the second image. Since the image acquisition process of the two image sensors is completed in one shot, the exposure degree is similar, so the resolution of the first image is higher, but affected by the amount of light, its noise may be more, the second image on the contrary.

Continuing to refer to FIG. 1, in step S120, an edge feature image is extracted from the first image.

Among them, the edge feature refers to the feature in the image where the feature (such as image texture, pixel gray level) is discontinuously distributed. An important difference between high-resolution images and low-resolution images is that high-resolution images have richer edge features. This exemplary embodiment preserves the detailed information in the image by extracting the edge feature image from the first image with the first resolution.

In an optional implementation manner, referring to FIG. 6, step S120 may be specifically implemented through the following steps S610 to S630:

Step S610, using Fourier transform to construct a noise separation function on the first image to obtain a denoising constraint item;

Step S620, construct a difference constraint item according to the gray gradient difference of the first image in the x direction and the y direction;

In step S630, a constraint function is established according to the denoising constraint item and the difference constraint item, and the edge feature image corresponding to the first image is obtained by solving the minimum value of the constraint function.

Among them, the first image itself carries certain noise information (noise points), which is embodied as a part of the edge features. When extracting the edge features, it is hoped to reduce the influence of noise. A noise-free image is sparsely similar, that is, it can be represented by fewer parameters or feature vectors, while the noise is irregular, non-similar, and non-sparse. Using this feature, a noise separation function can be constructed to obtain the edge Denoising constraints in extraction:

R(noise)=||F ^H DF(IMG _E -IMG1)||; (1)

Among them, R (noise) represents the denoising constraint item, F represents the matrix Fourier transform operator, H represents the conjugate transpose, D represents the transform domain, IMG _E represents the edge feature image (independent variable), IMG1 represents the first image . Since the energy of the noise is small and the gradient is relatively pure, a better denoising effect can be achieved by formula (1).

In a two-dimensional image, the x direction and the y direction represent the horizontal and vertical directions, respectively. The edge feature of the first image itself contains the components in these two directions. Therefore, by calculating the gray gradient difference in the x direction and the y direction, the difference constraint term can be constructed:

R(diff)=a ₁ ||G _x IMG _E ||+a ₂ ||G _y IMG _E ||; (2)

Among them, R(diff) represents the difference constraint item, a ₁ and a ₂ are the coefficients of two parts, which represent their respective weights, which are adjustable parameters, which can be adjusted according to experience and actual needs. G _x and G _y respectively represent the x direction And the difference operator in the y direction.

Adding formula (1) and formula (2) can establish a constraint function, which is used to indicate that the constraint condition is met during edge feature extraction. By solving the minimum value of the constraint function, the edge feature image corresponding to the first image can be obtained. As follows:

among them,

Is the first sub-constraint item, which is the L2 norm of the noise separation function; ||G _x IMG _E || ₁ is the second sub-constraint item, which is the L1 norm of the gray gradient difference of the first image in the x direction; ||G _y IMG _E || ₁ is the third sub-constraint item, which is the L1 norm of the gray gradient difference of the first image in the y direction; the three-part sub-constraint can be adjusted _{by setting the values of a 1} and a ₂ The weight ratio of the items meets actual needs. Equation (3) can be solved by iteration. When the constraint function reaches the convergence condition, the corresponding edge feature image is obtained.

Before extracting edge features, referring to Figure 7, grayscale processing can be performed on the first image. For example, all the pixel values of the first image can be mapped to [-127, +127] through the normalization process of removing the mean value. Within the range of, it can be converted into a standard gray-scale image, and then the gradient difference images of the gray-scale image in the x direction and the y direction are obtained respectively, which are used to calculate the difference operator in the subsequent edge feature extraction. In an optional implementation manner, referring to FIG. 8, step S620 may include the following steps S810 to S830:

Step S810: Perform grayscale processing on the first image;

Step S820: Obtain the gradient difference image in the x direction and the gradient difference image in the y direction of the first image.

Step S830, construct a difference constraint item based on the x-direction gradient difference image and the y-direction gradient difference image.

Among them, the x-direction gradient difference image and the y-direction gradient difference image can be separately calculated to construct the difference constraint terms in the two directions; or the x-direction gradient difference image and the y-direction gradient difference image can be synthesized first to obtain the inclusion A comprehensive gradient difference image of gray gradient information in two directions, and then a difference constraint item is constructed for the image, and the difference constraint can be realized through a sub-constraint item.

Continuing to refer to FIG. 1, in step S130, super-resolution reconstruction is performed on the second image to obtain a third image.

Among them, the third image is the first resolution. Super-resolution reconstruction refers to the reconstruction of high-resolution images from low-resolution images through interpolation and other methods.

In an optional implementation manner, when the image sensor collects images, it may collect multiple consecutive frames of second images. Then, using the first resolution as a reference, the super-resolution reconstruction algorithm is used to process the second images of consecutive multiple frames to obtain the third image of the first resolution. In the multi-frame second image, images are continuously collected for the same object, and each target in the image has a slight deviation in position. The super-resolution reconstruction algorithm can reconstruct the detailed image based on the information in the deviation. For example, referring to FIG. 9, step S130 may include the following steps S910 to S930:

In step S910, a second image of one frame is selected as a reference frame, and motion change parameters between the second image of other frames and the reference frame are calculated.

Step S920: Perform image registration on at least two frames of second images according to the aforementioned motion change parameters, and process the complementary non-redundant information between the at least two frames of second images to determine interpolation parameters. Image registration can use gradient image template matching alignment algorithm or feature operator-based alignment algorithm. In general, the more the number of second images selected during image registration, the more accurate the interpolation parameters obtained.

In step S930, the reference frame is interpolated by the above-mentioned interpolation parameters to obtain a third image of the first resolution.

It should be noted that after the interpolation is performed, post-processing such as blur and noise removal can also be performed to optimize the third image.

In an alternative embodiment, a multi-channel input convolutional neural network model can also be trained according to the number of frames of the second image, each channel is used to input a frame of low-resolution image, and the output of the model is the original high Resolution image, so that after multiple frames of second images are input to the model, the corresponding third image can be output.

The present disclosure does not limit which super-resolution reconstruction algorithm is specifically adopted.

Continuing to refer to FIG. 1, in step S140, the edge feature image and the third image are merged to obtain the final image.

Among them, when fusing the edge feature image and the third image, the edge feature image is mainly used to enhance the edge details in the third image, and the richer high-resolution detail information in the first image is incorporated into the third image.

In an embodiment, referring to FIG. 10, step S140 can be specifically implemented through the following steps S1010 to S1040:

Step S1010, define the independent variables of the final image;

Step S1020, construct a first sub-loss item according to the difference between the third image and the independent variable;

Step S1030, construct a second sub-loss item based on the difference between the edge feature image and the gray gradient difference of the independent variable in the x direction, and construct a second sub-loss item based on the difference between the edge feature image and the gray gradient difference of the independent variable in the y direction The third sub-loss item;

In step S1040, a loss function is established according to the first sub-loss item, the second sub-loss item and the third sub-loss item, and the final image is obtained by solving the minimum value of the loss function.

Among them, the loss function can be as follows:

Among them, IMG _F represents the final image (independent variable), and IMG3 represents the third image.

Is the first sub-loss item, which is a measure of the difference between the third image and the final image. It can use the L2 norm or other methods; ||G _x IMG _F -IMG _E || ₁ is the second sub-loss item , Is a measure of the difference between the gray gradient difference of the final image in the x direction and the edge feature image. The L1 norm can be used, or other methods can be used; ||G _y IMG _F -IMG _E || ₁ is the third The sub-loss item is a measure of the difference between the gray gradient difference of the final image in the y direction and the edge feature image. It can use the L1 norm or other methods; a ₃ and a ₄ are the second sub-loss items and The coefficients of the third sub-loss item represent their respective weights and are adjustable parameters. The value can be adjusted according to experience and actual needs to adjust the weight ratio of the three sub-loss items.

The above loss function actually expresses the error between the image fusion and the original image content. By solving the minimum value of the loss function, the original image information can be retained to the greatest extent, and the final image with the highest degree of restoration can be obtained.

In another embodiment, when fusing the third image and the edge feature image, each pixel in the edge feature image is traversed; for each pixel, if the gray value of the pixel in the edge feature image is 0, then The pixel value in the third image is used. If the gray value is not 0, the pixel value in the edge feature image is used. In this way, the edge feature information can also be incorporated into the third image to reconstruct detailed features.

After the final image is obtained, it can be directly output, for example, displayed in the user interface, or automatically saved to the corresponding folder.

In an optional implementation manner, the first resolution and the second resolution may be determined in advance according to the current exposure parameters. The current exposure parameters can include the current photosensitivity parameters, shutter parameters (shooting time), and lighting parameters of the surrounding environment, etc. The system can estimate the amount of light when taking pictures according to the current exposure parameters to determine the appropriate first resolution and second resolution. Resolution. Several specific implementations are provided below:

1. Fix the multiple relationship between the first resolution and the second resolution. For example, the first resolution is always 4 times the second resolution. When determining the two resolutions, only the appropriate second resolution needs to be calculated. The calculation relationship between the current exposure parameter and the second resolution can be pre-configured based on experience, for example, it can be a linear proportional relationship. The higher the current exposure parameter, the greater the second resolution, and then the second resolution is calculated.

2. The calculation relationship between the current exposure parameters and the second resolution can also be pre-configured. The first resolution is fixed to the maximum pixel of the image sensor. The second resolution is calculated according to the current exposure parameters during actual shooting, and only the second resolution is adjusted. Rate can be.

3. Calculate multiple sets of appropriate first resolution and second resolution according to the current exposure parameters, and display them on the shooting interface for the user to manually select. This method is particularly suitable for environments with unstable light.

The first resolution and the second resolution are determined in the above-mentioned manner to adapt to the current lighting conditions and exposure settings, etc., so that the quality of image shooting can be improved.

Fig. 11 shows the flow of this exemplary embodiment: first collect a first image IMG1 and a multi-frame second image IMG2; extract gradient difference images in the x direction and y direction from IMG1, and extract Edge feature image IMG _E ; use IMG2 for super-resolution reconstruction to obtain a third image IMG3, IMG3 and IMG1 have the same resolution, so the resolution is the same as _{IMG E} _{; finally IMG3 and IMG E} are fused, and the loss is established Function and solve the minimum value to get the final image IMG _F.

In summary, in this exemplary embodiment, a relatively high-resolution first image and a relatively low-resolution second image are collected, edge feature images are extracted from the first image, and the second image is super-resolution reconstructed into the first image. Three images, make the resolution of the third image and the first image the same, and finally merge the edge feature image and the third image to obtain the final image. Among them, the resolution of the first image is higher, but there may be more noise. Through edge feature extraction, the original detail information in the image is preserved; the resolution of the second image is lower, but the noise is less. After reconstruction into the third image, the third image has less noise, but there may be distortion in details; after fusing the two parts of the image, it is equivalent to incorporating the original image detail information into the low-noise image, which can achieve the first The advantages of the first image and the second image are complementary to solve the problem of high-pixel image noise and improve the quality of image shooting.

Exemplary embodiments of the present disclosure also provide an image fusion device, which can be configured in a terminal device equipped with an image sensor. Referring to FIG. 12, the image fusion apparatus 1200 may include a processor 1210 and a memory 1220; wherein, the memory 1220 stores the following program modules:

The image acquisition module 1221 is configured to acquire a first image and a second image collected by an image sensor, the first image is a first resolution, the second image is a second resolution, and the first resolution is higher than the second resolution;

The edge extraction module 1222 is used to extract edge feature images from the first image;

The image reconstruction module 1223 is configured to perform super-resolution reconstruction on the second image to obtain a third image, where the third image has the first resolution;

The fusion processing module 1224 is used for fusing the edge feature image and the third image to obtain the final image.

In an optional implementation manner, the above-mentioned image sensor includes a four-Bayer image sensor.

In an optional implementation manner, the image acquisition module 1221 may include:

The Bayer image acquisition unit is used to collect the original Bayer image based on the Four Bayer color filter array through the Four Bayer image sensor;

The first parsing unit is configured to perform demosaic processing and demosaic processing on the original Bayer image to obtain a first image, the first image being the first resolution;

The second analysis unit is used to merge four adjacent pixels of the same color in the original Bayer image into one pixel, and perform demosaicing processing on the Bayer image after the merged pixels to obtain a second image, which is the second resolution Rate; where the first resolution is four times the second resolution.

In an optional implementation manner, the above-mentioned image sensor includes a first image sensor and a second image sensor, and the pixels of the first image sensor are higher than those of the second image sensor. The image acquisition module 1221 may be used to acquire the first image acquired by the first image sensor and the second image acquired by the second image sensor.

In an optional implementation manner, the edge extraction module 1222 may include:

The denoising constraint unit is used to construct a noise separation function on the first image using Fourier transform to obtain a denoising constraint item;

The difference constraint unit is used to construct a difference constraint item according to the gray gradient difference of the first image in the x direction and the y direction;

The constraint function solving unit is used to establish a constraint function according to the denoising constraint item and the difference constraint item, and obtain the edge feature image corresponding to the first image by solving the minimum value of the constraint function.

In an optional implementation manner, the aforementioned denoising constraint items may include:

The first sub-constraint item is the L2 norm of the noise separation function.

In an optional implementation manner, the above-mentioned differential constraint item may include:

The second sub-constraint item is the L1 norm of the gray gradient difference of the first image in the x direction;

The third sub-constraint item is the L1 norm of the gray gradient difference of the first image in the y direction.

In an optional implementation manner, the differential constraint unit can be used for:

Perform grayscale processing on the first image;

Acquiring a gradient difference image in the x direction and a gradient difference image in the y direction of the first image;

The difference constraint term is constructed based on the x-direction gradient difference image and the y-direction gradient difference image.

In an optional implementation manner, the second image may include multiple consecutive frames of the second image.

In an optional implementation manner, the image reconstruction module 1223 may be used to process the second image of multiple consecutive frames using a super-resolution reconstruction algorithm to obtain a third image.

In an optional implementation manner, the image reconstruction module 1223 may be used for:

Select one frame of the second image as the reference frame, and calculate the motion change parameters between the second image of other frames and the reference frame;

Performing image registration on at least two frames of second images according to the aforementioned motion change parameters, and processing complementary non-redundant information between the at least two frames of second images to determine interpolation parameters;

The reference frame is interpolated through the above-mentioned interpolation parameters to obtain a third image of the first resolution.

In an optional implementation manner, the image reconstruction module 1223 may be used to:

The multi-channel input convolutional neural network model is used to process the above-mentioned continuous multiple frames of the second image, and output the corresponding third image.

In an optional implementation manner, the fusion processing module 1224 may include:

The independent variable definition unit is used to define the independent variables of the final image;

The sub-loss construction unit is used to construct the first sub-loss item according to the difference between the third image and the independent variable, and construct the second sub-loss item according to the difference between the edge feature image and the difference in the gray gradient of the independent variable in the x direction, And according to the difference between the edge feature image and the gray gradient difference of the independent variable in the y direction, construct the third sub-loss item;

The loss function solving unit is used to establish a loss function according to the first sub-loss item, the second sub-loss item and the third sub-loss item, and obtain the final image by solving the minimum value of the loss function.

In an optional implementation manner, the first sub-loss item adopts the L2 norm, and the second sub-loss item and the third sub-loss item adopt the L1 norm.

In an optional implementation manner, the fusion processing module 1224 may be used to:

Traverse each pixel in the edge feature image, if the gray value of the pixel is 0, the pixel value in the third image is used, if the gray value of the pixel is not 0, then the pixel value in the edge feature image is used .

In an optional implementation manner, the image acquisition module 1221 may also be used to determine the first resolution and the second resolution according to the current exposure parameters.

In an optional embodiment, the current exposure parameter includes at least one of the current light-sensing parameter, the shutter parameter, and the ambient lighting parameter.

The exemplary embodiments of the present disclosure also provide another image fusion device, which can be configured in a terminal device equipped with an image sensor. Referring to FIG. 13, the image fusion apparatus 1300 may include:

The image acquisition module 1310 is configured to acquire a first image and a second image collected by an image sensor, the first image is a first resolution, the second image is a second resolution, and the first resolution is higher than the second resolution;

The edge extraction module 1320 is used to extract edge feature images from the first image;

The image reconstruction module 1330 is configured to perform super-resolution reconstruction on the second image to obtain a third image, where the third image has the first resolution;

The fusion processing module 1340 is used for fusing the edge feature image and the third image to obtain a final image.

In an optional implementation manner, the image acquisition module 1310 may include:

In an optional implementation manner, the above-mentioned image sensor includes a first image sensor and a second image sensor, and the pixels of the first image sensor are higher than those of the second image sensor. The image acquisition module 1310 may be used to acquire the first image acquired by the first image sensor and the second image acquired by the second image sensor.

In an optional implementation manner, the edge extraction module 1320 may include:

The first sub-constraint item is the L2 norm of the noise separation function.

Perform grayscale processing on the first image;

In an optional implementation manner, the image reconstruction module 1330 may be used to process the second image of multiple consecutive frames using a super-resolution reconstruction algorithm to obtain a third image.

In an optional implementation manner, the image reconstruction module 1330 may be used to:

In an optional implementation manner, the fusion processing module 1340 may include:

In an optional implementation manner, the fusion processing module 1340 may be used to:

In an optional implementation manner, the image acquisition module 1310 may also be used to determine the first resolution and the second resolution according to the current exposure parameters.

The specific details of the modules/units in the above-mentioned device 1200 and device 1300 have been described in detail in the method part of the implementation. For undisclosed details, please refer to the method part of the implementation content, and therefore will not be repeated.

Exemplary embodiments of the present disclosure also provide a computer-readable storage medium, which can be implemented in the form of a program product, which includes program code. When the program product runs on a terminal device, the program code is used to make the terminal device Perform the steps according to various exemplary embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of this specification. The program product can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, the readable storage medium can be any tangible medium that contains or stores a program, and the program can be used by or in combination with an instruction execution system, device, or device.

The program product can adopt any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or a combination of any of the above. More specific examples (non-exhaustive list) of readable storage media include: electrical connections with one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable Type programmable read only memory (EPROM or flash memory), optical fiber, portable compact disk read only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.

The computer-readable signal medium may include a data signal propagated in baseband or as a part of a carrier wave, and readable program code is carried therein. This propagated data signal can take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. The readable signal medium may also be any readable medium other than a readable storage medium, and the readable medium may send, propagate, or transmit a program for use by or in combination with the instruction execution system, apparatus, or device.

The program code contained on the readable medium can be transmitted by any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

The program code used to perform the operations of the present disclosure can be written in any combination of one or more programming languages. The programming languages include object-oriented programming languages-such as Java, C++, etc., as well as conventional procedural programming. Language-such as "C" language or similar programming language. The program code can be executed entirely on the user's computing device, partly on the user's device, executed as an independent software package, partly on the user's computing device and partly executed on the remote computing device, or entirely on the remote computing device or server Executed on. In the case of a remote computing device, the remote computing device can be connected to a user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (for example, using Internet service providers). Business to connect via the Internet).

Exemplary embodiments of the present disclosure also provide a terminal device capable of implementing the above method. The terminal device 1400 according to this exemplary embodiment of the present disclosure will be described below with reference to FIG. 14. The terminal device 1400 shown in FIG. 14 is only an example, and should not bring any limitation to the function and scope of use of the embodiments of the present disclosure.

As shown in FIG. 14, the terminal device 1400 may be represented in the form of a general-purpose computing device. The components of the terminal device 1400 may include, but are not limited to: at least one processing unit 1410, at least one storage unit 1420, a bus 1430 connecting different system components (including the storage unit 1420 and the processing unit 1410), a display unit 1440, and an image sensor 1470. The sensor 1470 is used to collect images.

The storage unit 1420 stores program codes, and the program codes can be executed by the processing unit 1410, so that the processing unit 1410 executes the steps according to various exemplary embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of this specification. For example, the processing unit 1410 may execute any one or more of the method steps in FIG. 1, FIG. 3, FIG. 6, FIG. 8, FIG. 9, or FIG. 10.

The storage unit 1420 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 1421 and/or a cache storage unit 1422, and may further include a read-only storage unit (ROM) 1423.

The storage unit 1420 may also include a program/utility tool 1424 having a set of (at least one) program modules 1425. Such program modules 1425 include but are not limited to: an operating system, one or more application programs, other program modules, and program data, Each of these examples or some combination may include the implementation of a network environment.

The bus 1430 may represent one or more of several types of bus structures, including a storage unit bus or a storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local area using any bus structure among multiple bus structures bus.

The terminal device 1400 may also communicate with one or more external devices 1500 (such as keyboards, pointing devices, Bluetooth devices, etc.), and may also communicate with one or more devices that enable the user to interact with the terminal device 1400, and/or communicate with Any device (such as a router, modem, etc.) that enables the terminal device 1400 to communicate with one or more other computing devices. This communication can be performed through an input/output (I/O) interface 1450. In addition, the terminal device 1400 may also communicate with one or more networks (for example, a local area network (LAN), a wide area network (WAN), and/or a public network, such as the Internet) through the network adapter 1460. As shown in the figure, the network adapter 1460 communicates with other modules of the terminal device 1400 through the bus 1430. It should be understood that although not shown in the figure, other hardware and/or software modules can be used in conjunction with the terminal device 1400, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives And data backup storage system, etc.

Through the description of the above embodiments, those skilled in the art can easily understand that the example embodiments described here can be implemented by software, or can be implemented by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, U disk, mobile hard disk, etc.) or on the network , Including several instructions to make a computing device (which may be a personal computer, a server, a terminal device, or a network device, etc.) execute the method according to the exemplary embodiment of the present disclosure.

In addition, the above-mentioned drawings are merely schematic illustrations of the processing included in the method according to the exemplary embodiment of the present disclosure, and are not intended for limitation. It is easy to understand that the processing shown in the above drawings does not indicate or limit the time sequence of these processings. In addition, it is easy to understand that these processes can be executed synchronously or asynchronously in multiple modules, for example.

It should be noted that although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to exemplary embodiments of the present disclosure, the features and functions of two or more modules or units described above may be embodied in one module or unit. Conversely, the features and functions of a module or unit described above can be further divided into multiple modules or units to be embodied.

Those skilled in the art can understand that various aspects of the present disclosure can be implemented as a system, a method, or a program product. Therefore, various aspects of the present disclosure can be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "Circuit", "Module" or "System". After considering the specification and practicing the invention disclosed herein, those skilled in the art will easily think of other embodiments of the present disclosure. This application is intended to cover any variations, uses, or adaptive changes of the present disclosure. These variations, uses, or adaptive changes follow the general principles of the present disclosure and include common knowledge or conventional techniques in the technical field that are not disclosed in the present disclosure. means. The description and the embodiments are only regarded as exemplary, and the true scope and spirit of the present disclosure are pointed out by the claims.

It should be understood that the present disclosure is not limited to the precise structure that has been described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is limited only by the appended claims.

Claims

An image fusion method applied to a terminal device equipped with an image sensor, characterized in that the method includes:

Acquire a first image and a second image collected by the image sensor, the first image is a first resolution, the second image is a second resolution, and the first resolution is higher than the second Resolution

Extracting an edge feature image from the first image;

Performing super-resolution reconstruction on the second image to obtain a third image, the third image being the first resolution;

The edge feature image and the third image are fused to obtain a final image.
The method of claim 1, wherein the image sensor comprises a four-Bayer image sensor.
The method according to claim 2, wherein the obtaining the first image and the second image collected by the four Bayer image sensor comprises:

Collecting the original Bayer image based on the four Bayer color filter array through the four Bayer image sensor;

Performing demosaic processing and demosaic processing on the original Bayer image to obtain the first image, the first image being the first resolution;

Combine four adjacent pixels of the same color in the original Bayer image into one pixel, and perform demosaicing processing on the combined Bayer image to obtain the second image. The second image is the second Resolution

Wherein, the first resolution is four times the second resolution.
The method according to claim 1, wherein the image sensor comprises a first image sensor and a second image sensor, and the pixels of the first image sensor are higher than those of the second image sensor;

The acquiring the first image and the second image collected by the image sensor includes:

Acquire a first image collected by the first image sensor and a second image collected by the second image sensor.
The method according to claim 1, wherein the extracting an edge feature image from the first image comprises:

Using Fourier transform to construct a noise separation function on the first image to obtain a denoising constraint item;

Construct a difference constraint term according to the gray gradient difference of the first image in the x direction and the y direction;

A constraint function is established according to the denoising constraint item and the difference constraint item, and an edge feature image corresponding to the first image is obtained by solving the minimum value of the constraint function.
The method according to claim 5, wherein the denoising constraint item comprises:

The first sub-constraint item is the L2 norm of the noise separation function.
The method according to claim 5, wherein the difference constraint item comprises:

The second sub-constraint item is the L1 norm of the gray gradient difference of the first image in the x direction; and

The third sub-constraint item is the L1 norm of the gray gradient difference of the first image in the y direction.
The method according to claim 5, wherein the constructing a difference constraint item according to the gray gradient difference of the first image in the x direction and the y direction comprises:

Performing grayscale processing on the first image;

Acquiring an x-direction gradient difference image and a y-direction gradient difference image of the first image;

The difference constraint term is constructed according to the x-direction gradient difference image and the y-direction gradient difference image.
The method according to claim 1, wherein the second image comprises a second image of multiple consecutive frames.
The method according to claim 9, wherein the super-resolution reconstruction of the second image to obtain a third image comprises:

The super-resolution reconstruction algorithm is used to process the second image of the continuous multiple frames to obtain the third image.
The method according to claim 10, wherein said using a super-resolution reconstruction algorithm to process the second image of the continuous multiple frames to obtain the third image comprises:

Selecting a second image of a frame as a reference frame, and calculating motion change parameters between the second image of other frames and the reference frame;

Performing image registration on at least two frames of second images according to the motion change parameters, and processing complementary non-redundant information between the at least two frames of second images to determine interpolation parameters;

Interpolate the reference frame by using the interpolation parameter to obtain the third image of the first resolution.
The method according to claim 10, wherein said using a super-resolution reconstruction algorithm to process the second image of the continuous multiple frames to obtain the third image comprises:

The convolutional neural network model of multi-channel input is used to process the second image of consecutive multiple frames, and output the corresponding third image.
The method according to claim 1, wherein the fusing the edge feature image and the third image to obtain a final image comprises:

Define the independent variables of the final image;

Construct a first sub-loss item according to the difference between the third image and the independent variable;

Construct a second sub-loss item according to the difference between the edge feature image and the gray gradient difference of the independent variable in the x direction, and construct the second sub-loss item, and according to the edge feature image and the gray gradient of the independent variable in the y direction The difference of the difference, construct the third sub-loss item;

A loss function is established according to the first sub-loss item, the second sub-loss item, and the third sub-loss item, and the final image is obtained by solving the minimum value of the loss function.
The method according to claim 13, wherein the first sub-loss item adopts an L2 norm, and the second sub-loss item and the third sub-loss item adopt an L1 norm.
The method according to claim 1, wherein the fusing the edge feature image and the third image to obtain a final image comprises:

Traverse each pixel in the edge feature image, if the gray value of the pixel is 0, the pixel value in the third image is used, and if the gray value of the pixel is not 0, then The pixel value in the edge feature image is used.
The method according to any one of claims 1 to 15, wherein the method further comprises:

The first resolution and the second resolution are determined according to the current exposure parameters.
The method according to claim 16, wherein the current exposure parameter comprises at least one of a current photosensitive parameter, a shutter parameter, and an ambient light parameter.
An image fusion device configured in a terminal device equipped with an image sensor, characterized in that the device includes a processor; wherein the processor is used to execute the following program modules stored in a memory:

An image acquisition module for acquiring a first image and a second image collected by the image sensor, the first image is a first resolution, the second image is a second resolution, and the first resolution Higher than the second resolution;

An edge extraction module for extracting edge feature images from the first image;

An image reconstruction module, configured to perform super-resolution reconstruction on the second image to obtain a third image, the third image being the first resolution;

The fusion processing module is used for fusing the edge feature image and the third image to obtain a final image.
A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 17 when the computer program is executed by a processor.
A terminal device, characterized in that it comprises:

processor;

A memory for storing executable instructions of the processor; and

Image Sensor;

Wherein, the processor is configured to execute the method according to any one of claims 1 to 17 by executing the executable instructions.