WO2019105261A1

WO2019105261A1 - Background blurring method and apparatus, and device

Info

Publication number: WO2019105261A1
Application number: PCT/CN2018/116475
Authority: WO
Inventors: 欧阳丹; 谭国辉
Original assignee: Oppo广东移动通信有限公司
Priority date: 2017-11-30
Filing date: 2018-11-20
Publication date: 2019-06-06
Also published as: CN108053363A

Abstract

Disclosed in the present application are a background blurring method and apparatus, and a device. The method comprises: obtaining a main image captured by a main camera and an auxiliary image captured by an auxiliary camera, and obtaining depth of field information of the main image according to the main image and the auxiliary image; determining original blurring intensities of different subregions in a background region of the main image according to the depth of field information and a focus region; determining distribution orientations of the different subregions according to a display mode of the main image, and determining blurring weights of the different subregions according to a weight setting strategy corresponding to the distribution orientations; determining target blurring intensities of the different subregions according to the original blurring intensities and the corresponding blurring weights of the different subregions; and blurring the background region of the main image according to the target blurring intensities of the different subregions. Therefore, the blurring effect is more natural, and more approaches to a real optical defocusing effect.

Description

Background blur processing method, device and device

Cross-reference to related applications

This application claims the priority of the Chinese Patent Application No. "201711242134.4" filed on November 30, 2017 by Guangdong Opal Mobile Communications Co., Ltd., which is entitled "Background Blurring Processing Method, Apparatus and Equipment".

Technical field

The present application relates to the field of electronic devices, and in particular, to a background blur processing method, apparatus, and terminal device.

Background technique

With the advancement of terminal device manufacturing technology such as smart phones, more terminal devices currently use a dual camera system, that is, two depth images acquired by two cameras are used to calculate depth information, and then use depth information to perform blurring processing, wherein In the process of blurring, the user usually requires the effect of blurring to be closer to the true optical virtual focus effect, that is, the greater the depth of field, the higher the intensity of blurring. However, the current depth of field calculation accuracy is limited, and may not be accurately calculated beyond The depth of field of the distance, and thus, the blurring of the far distance region in the image cannot be achieved according to the depth of field, resulting in a poor visual effect of the blurring process.

Application content

The present invention provides a background blur processing method, device and device, so as to solve the problem that the depth of field beyond a certain distance cannot be accurately calculated in the prior art, and thus the darkening of the image in the image cannot be realized according to the depth of field, resulting in virtual Technical problems with poor visual effects.

The embodiment of the present application provides a background blurring processing method, including: acquiring a main image captured by a main camera and a sub image captured by a sub camera, and acquiring depth information of the main image according to the main image and the sub image; Determining an original blurring intensity of different sub-regions in the background area of the main image according to the depth information and the in-focus area; determining a distribution orientation of the different sub-areas according to a display manner of the main image, and according to the distribution orientation Corresponding weight setting policy determines a blurring weight of the different sub-regions; determining a target blurring strength of the different sub-regions according to the original blurring intensity of the different sub-regions and corresponding blurring weights; The target blurring intensity of the region blurs the background area of the main image.

Another embodiment of the present invention provides a background blur processing apparatus, including: a first acquiring module, configured to acquire a main image captured by a main camera and a sub image captured by a sub camera, and according to the main image and the sub image Acquiring the depth information of the main image; the first determining module is configured to determine an original blurring intensity of different sub-regions in the background area of the main image according to the depth information and the focus area; and a second determining module, configured to Determining a distribution orientation of the different sub-regions, and determining a blurring weight of the different sub-regions according to a weight setting policy corresponding to the distribution orientation; and a third determining module, configured to determine, according to the different The original blurring intensity of the sub-region and the corresponding blurring weight determine the target blurring intensity of the different sub-regions; the processing module is configured to perform virtualizing on the background area of the main image according to the target blurring intensity of the different sub-regions Processing.

A further embodiment of the present application provides a computer device including a memory and a processor, wherein the memory stores computer readable instructions, and when the instructions are executed by the processor, the processor performs the above implementation of the present application. The background blurring method described in the example.

A further embodiment of the present application provides a non-transitory computer readable storage medium having a computer program stored thereon, which when executed by a processor, implements a background blurring processing method as described in the above embodiments of the present application.

The technical solutions provided by the embodiments of the present application may include the following beneficial effects:

Obtaining a main image captured by the main camera and a sub-image captured by the sub-camera, and acquiring depth information of the main image according to the main image and the sub-image, and determining an original blurring intensity of different sub-regions in the background area of the main image according to the depth information and the focus area, Determining the distribution orientation of different sub-areas according to the display mode of the main image, and determining the blurring weights of different sub-areas according to the weight setting strategy corresponding to the distribution orientation, and further, according to the original blurring intensity of the different sub-areas and the corresponding blurring weights The target blurring intensity of different sub-areas is determined. Finally, the main image background area is blurred according to the target blurring intensity of different sub-areas. Thereby, the effect of the blurring effect is more natural and closer to the real optical virtual focus.

DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings to be used in the embodiments will be briefly described below. It is obvious that the drawings in the following description are some embodiments of the present disclosure, Those skilled in the art can also obtain other drawings based on these drawings without paying any creative work.

1 is a flow chart of a background blurring processing method according to an embodiment of the present application;

2 is a schematic diagram of the principle of triangulation according to an embodiment of the present application;

FIG. 3 is a schematic diagram of a dual camera viewing angle coverage according to an embodiment of the present application; FIG.

4 is a schematic diagram of acquiring a depth of field of a dual camera according to an embodiment of the present application;

FIG. 5(a) is a schematic diagram showing division of a plurality of sub-areas in a background area of a main image according to an embodiment of the present application;

FIG. 5(b) is a schematic diagram showing division of a plurality of sub-areas in a background area of a main image according to another embodiment of the present application;

6 is a flow chart of a background blur processing method according to an embodiment of the present application;

FIG. 7 is a schematic structural diagram of a background blur processing apparatus according to an embodiment of the present application; FIG.

FIG. 8 is a schematic structural diagram of a background blur processing apparatus according to another embodiment of the present application; FIG.

9 is a schematic structural diagram of a background blurring processing apparatus according to still another embodiment of the present application;

FIG. 10 is a schematic diagram of an image processing circuit according to another embodiment of the present application.

Detailed ways

The embodiments of the present application are described in detail below, and the examples of the embodiments are illustrated in the drawings, wherein the same or similar reference numerals are used to refer to the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are intended to be illustrative, and are not to be construed as limiting.

The background blur processing method, apparatus and terminal device of the embodiment of the present application are described in detail below with reference to the accompanying drawings.

The execution body of the background blur processing method and apparatus of the embodiment of the present application may be a terminal device, where the terminal device may be a hardware device with a dual camera such as a mobile phone, a tablet computer, a personal digital assistant, a wearable device, or the like. The wearable device can be a smart bracelet, a smart watch, smart glasses, and the like.

Based on the above analysis, in the prior art, since the blurring is performed according to the depth of field information, when the depth of field information cannot be accurately obtained due to the limitation of accuracy, the blurring intensity of the corresponding region cannot be achieved, thereby affecting the image. Blurring effect.

In order to solve the above technical problem, the present application provides a background blurring processing method for controlling the positional relationship of regions corresponding to different depths of field with different blur depths, and controlling regions corresponding to different depths of field to perform blurring of different intensities, thereby Accurately obtaining the depth of field information can also make the area corresponding to different depths of field get the appropriate intensity blur.

FIG. 1 is a flowchart of a background blur processing method according to an embodiment of the present application. As shown in FIG. 1, the method includes the following steps:

Step 101: Acquire a main image captured by the main camera and a sub image captured by the sub camera, and acquire depth information of the main image according to the main image and the sub image.

Wherein, after focusing on the subject being photographed, the spatial depth of the clear imaging allowed by the human eye before and after the focus area where the subject is located is the depth of field.

It should be noted that in practical applications, the depth of field of the human eye is mainly determined by binocular vision to distinguish the depth of field. This is the same as the principle of dual camera resolution of depth of field, mainly relying on the principle of triangular ranging as shown in Figure 2. Based on Figure 2, in the actual space, the imaged object is drawn, as well as the positions of the two cameras O _R and O _T , and the focal planes of the two cameras. The focal plane is at a distance f from the plane of the two cameras. Two cameras are imaged at the focal plane position to obtain two captured images.

Among them, P and P' are the positions of the same object in different captured images, respectively. The distance from the P point to the left boundary of the captured image is X _R , and the distance from the P′ point to the left boundary of the captured image is X _T . O _R and O _T are two cameras respectively, and the two cameras are on the same plane with a distance B.

Based on the principle of triangulation, the distance Z between the object in Figure 2 and the plane of the two cameras has the following relationship:

Based on this, you can push

Where d is the difference in distance between the positions of the same object in different captured images. Since B and f are constant values, the distance Z of the object can be determined according to d.

It should be emphasized that the above formula is implemented based on two parallel cameras. However, there are actually many problems when actually used. For example, in the above picture, there are always some scenes in the depth of field of the camera that cannot intersect, so the actual For the depth of field calculation, the FOV design of the two cameras will be different. The main camera is used to take the main image of the actual image. The sub-image obtained by the sub-camera is mainly used to calculate the depth of field. Based on the above analysis, the sub-camera The FOV is generally larger than the main camera, but even if it is as shown in Figure 3, objects that are closer together may still be in different images acquired by the two cameras. The adjusted calculated depth of field range is as follows:

You can calculate the depth of field of the main image based on the adjusted formula.

Of course, in addition to the triangulation method, other methods can be used to calculate the depth of field of the main image. For example, when the main camera and the sub camera are photographed for the same scene, the distance between the object in the scene and the camera is imaged by the main camera and the sub camera. The displacement difference, the posture difference, and the like are proportional, and therefore, in one embodiment of the present application, the above-described distance Z can be obtained according to such a proportional relationship.

For example, as shown in FIG. 4, a map of different point differences is calculated by the main image acquired by the main camera and the sub-image acquired by the sub-camera, and is represented by a disparity map, which is the same on the two graphs. The difference in displacement of the points, but since the difference in displacement in the triangulation is proportional to Z, the disparity map is often used directly as the depth of field map.

Step 102: Determine an original blurring intensity of different sub-regions in the background area of the main image according to the depth of field information and the focus area.

It can be understood that since the range of imaging before the focus area is the foreground depth of field, the area corresponding to the foreground depth of field is the foreground area, the range of clear imaging after the focus area is the background depth of field, and the area corresponding to the background depth of field is the background area, according to the depth of field information and focus The area determines the background area of the main image, and further, the original blurring intensity of the different sub-areas in the background area is initially determined, and the original blurring intensity is used as an adjustment basis for subsequent blurring of each sub-area of the background area.

As a possible implementation, the background area is divided into different sub-areas in a horizontal direction (parallel to the direction of the focus area).

Determining, according to the depth information and the focus area, first depth information of the foreground area in the main image and second depth information of the background area, and acquiring average depth information of different sub-areas in the background area of the main image according to the second depth information, and further, according to the first The depth of field information and the average depth of field information of different sub-areas obtain the original blurring intensity of different sub-areas.

Specifically, in this example, the background area is divided into a plurality of different sub-areas according to the horizontal direction, wherein the size and shape of each sub-area can be adjusted according to application requirements, and the size and shape of the plurality of different sub-areas can be Similarly, the difference may be different. Further, the depth of field of the closest position from the focus area in each sub-area and the depth of field farthest from the focus area are respectively obtained, and the average depth of field of the closest position and the depth of field of the farthest position are averaged. The average depth of field is taken as the average depth information of the corresponding sub-area, and the original blurring intensity is determined according to the average depth information of the sub-area, wherein the higher the average depth information, the greater the original blurring intensity.

The depth of field interval of each sub-area can be adjusted according to application requirements. The depth of field interval of multiple different sub-areas can be the same or different. As shown in FIG. 5( a ), multiple sub-areas can be shaped according to the background area. The sub-areas are divided into a plurality of sub-areas that are inconsistent in size, or, for the convenience of further blurring processing, as shown in FIG. 5(b), the plurality of sub-areas may be horizontally distributed from the bottom to the top of the image having the same width and image width.

It should be emphasized that if the depth of field of the closest position of the sub-area from the focus area and the depth of field farthest from the focus area are not obtained due to the limitation of the accuracy of the depth of field calculation, the depth information obtained in the sub-area may be obtained. The depth of field probability distribution in the middle is calculated as the average depth of field of the sub-area.

Step 103: Determine a distribution orientation of different sub-regions according to a display manner of the main image, and determine a blur weight of the different sub-regions according to a weight setting policy corresponding to the distribution orientation.

The display manner includes the display reverse direction of the shooting subject, the proportion of the entire image, and the like.

It can be understood that, in the first photographing scene, the bottom of the image is the ground, the top of the image is the sky, and the subject of the photograph is located on the ground. Therefore, the sub-region in the background area of the main image is closer to the top of the image. The more it is not related to the current subject, the closer it is to the bottom of the image, the more relevant it is to the current subject. Or, in the second photographing scene, the right side of the image is a portrait, and the left side of the image is a beach or a sea, etc., if the current photographing mode is the portrait mode, the closer to the left side of the image, the more the current subject is unrelated. The closer to the right side of the image, the more relevant it is to the subject being photographed, or, in the third photographing scene, that is, in the portrait photographing mode, the portrait is occupied in a larger proportion in the entire image, and the background area is only in the image. The more the four corner positions, the farther away from the area where the portrait is located, the more the current subject is irrelevant.

Therefore, in the embodiment of the present application, the distribution orientation of different sub-regions may be determined according to the display manner of the main image, and further, the blurring weights of different sub-regions may be determined according to the weight setting strategy corresponding to the distribution orientation, for example, for the above In a scene, the sub-region closer to the upper direction is not related to the subject in the main image, and the sub-region closer to the lower direction is related to the subject in the main image, thereby determining according to the top-down weight reduction strategy. The blurring weight of different sub-areas, wherein the up and down direction does not only include the upper and lower sides of the traditional physical, referring to the above analysis, it is related to the display mode of the main image, and if the display mode of the main image is up and down, the The upper and lower directions indicate physical upper and lower, and if the display mode of the main image is horizontal display, the upper and lower directions indicate the left and right direction.

For example, when the display mode of the main image is up and down, and the main body of the photograph is relatively small relative to the whole image, and the upper and lower orientations indicate the upper and lower physics, the sub-regions closer to the top of the image have greater blurring weights. The sub-area near the bottom has a small blurring weight, which makes the final blur effect more natural and closer to the real optical virtual focus effect.

In actual applications, different display modes may be used to estimate the display mode of the main image according to different application scenarios. For example, the orientation of the terminal device may be obtained by detecting the gyroscope information of the terminal device, and then the terminal device is inferred to take a picture. The way the main image is displayed.

In the implementation of the present application, there are various ways to determine the blurring weights of different sub-areas according to the weight setting strategy corresponding to the distributed orientation. For example, the relationship between the distribution orientation and the blurring effect may be obtained according to a large number of experiments in advance, and further, according to the relationship The correspondence between the distribution orientation and the weight setting policy is established and stored, so that after determining the display manner of the main image, the corresponding relationship is queried to determine a corresponding weight setting policy.

For a clearer explanation, the following is an example of determining the ambiguity weight of different sub-areas according to the top-to-bottom weight reduction strategy:

method one:

In this mode, a linear distribution curve including a mapping relationship between the positioning coordinates of the sub-region and the blurring weight, or a nonlinear distribution curve, wherein the positioning coordinate can be any point coordinate of the central region of the sub-region, is further set in advance. The positioning coordinates of different sub-areas are determined according to the positioning coordinate query according to a preset linear weight distribution curve or a nonlinear weight distribution curve to determine the blurring weights of different sub-areas.

Method 2:

According to a large amount of experimental data, the correspondence between the display direction of the image and the blurring weight of the sub-regions of different orientations is constructed, and a deep neural network model is constructed according to the learning result, thereby inputting the display direction of the main image and the orientation of different sub-regions in the model. Further, the blurring weight of the corresponding sub-region is obtained according to the output of the model.

Step 104: Determine target blurring strengths of different sub-regions according to original blurring strengths of different sub-regions and corresponding blurring weights.

Step 105: Perform blurring processing on the background area of the main image according to the target blurring intensity of different sub-areas.

Specifically, after determining the original blurring intensity and the corresponding blurring weight of different sub-regions, determining the target blurring intensity of different sub-regions according to the original blurring intensity of the different sub-regions and the corresponding blurring weight, and according to different sub-regions The target blurring intensity of the region blurs the background area of the main image. Therefore, according to the relationship between the orientation of the sub-region and the display direction of the main image, different sub-regions are blurred by different intensities, and it is not necessary to know each sub-region. The precise depth of field information can also make the different sub-regions get the corresponding intensity blur, making the blurring result more natural.

That is to say, the original blurring intensity of different sub-regions determined only according to the initially obtained depth of field information may be deviated due to the inaccuracy of the depth of field information acquisition, and the blurring intensity corresponding to the real optical virtual focal effect may be deviated. In the embodiment of the present application, the blurring weights of different sub-regions are further determined according to the display direction of the main image and the orientations of different sub-regions, and then the original blurring strength is corrected according to the blurring weight to determine the target blurring intensity, so that the virtual The result is more natural.

It should be noted that, depending on the application scenario, the original blurring strength of the different sub-regions and the corresponding blurring weights are different in determining the target blurring strength of different sub-regions, including but not limited to the following ways:

method one:

By obtaining the product of the original blurring intensity of different sub-regions and the corresponding blurring weight, the target blurring strength of different sub-regions is determined. For example, the blurring weights of the two sub-regions whose original blurring intensities are a and b respectively are 80% and 90%, then a*80% and b*90% can be used as the target blurring intensity for the above two sub-regions, respectively.

Method 2:

In order to make the blurring effect of the adjacent sub-regions more natural, the blurring weights of the adjacent sub-regions with large differences in the original blurring intensity are subjected to a certain degree of close processing, for example, for adjacent sub-regions 1 and 2 If the original blurring intensity of sub-region 1 is larger than that of sub-region 2, then the blurring weight of sub-region 1 is multiplied by a coefficient smaller than 1, wherein the original blurring intensity of sub-region 1 is compared The larger the sub-region 2 is, the smaller the coefficient is. Further, the product of the original blurring intensity, the blurring weight and the coefficient of the sub-region 1 is taken as the target blurring intensity of the sub-region 1, and the original blurring intensity of the sub-region 2 is obtained. The product of the divisor weight is used as the target blur strength of sub-region 2, or the blur weight of sub-region 2 is multiplied by a coefficient greater than 1, wherein the original blur strength of sub-region 1 is compared with sub-region 2 The larger the coefficient is, the larger the product of the original blurring intensity, the blurring weight and the coefficient of the sub-region 2 is taken as the target blurring intensity of the sub-region 2, and the original blurring intensity and the blurring weight of the sub-region 1 are Product of the sub-region 1 as the target blur Degree.

Further, the manner of performing the blurring process on the background area of the main image according to the target blurring intensity of different sub-areas also includes, but is not limited to, the following processing manners:

As a possible implementation:

Determining the blurring coefficient of each pixel in the background region according to the target blurring intensity of different sub-regions and the depth information of each pixel in different sub-regions, and performing Gaussian blur processing on the background region according to the blurring coefficient of each pixel in the background region By blurring the photo, the effect that the background depth information is larger and the degree of blurring is stronger is achieved.

Of course, in this embodiment, if the depth information of each pixel cannot be accurately obtained, the depth information of the sub-area may be acquired in the manner described in step 102.

For a clearer description, the background blurring processing manner of the present application is exemplified below with reference to a specific application scenario:

As shown in FIG. 6, acquiring a main image captured by the main camera and a sub image captured by the sub camera, and acquiring depth information of the main image according to the main image and the sub image, and further, according to the depth information and focus The region determines the original blurring intensity of different sub-regions in the background area of the main image, multiplies the original blurring intensity by the blurring weight of the sub-region where the original blurring intensity is located, and obtains the target blurring intensity of the corresponding sub-region, Further, the background area of the main image is blurred according to the target blurring intensity to obtain a blurred image.

With reference to FIG. 6 , the display direction of the main image may be estimated according to the gyroscope information of the terminal device, and then the upper and lower orientations of the different sub-areas are determined according to the display direction of the main image and determined according to the top-to-bottom weight reduction strategy. The ambiguity weight of different sub-areas.

Based on the above embodiment, in addition to dividing the background region into sub-regions in a direction parallel to the in-focus region, the background region may be divided into different sub-regions in a vertical direction (a direction perpendicular to the in-focus region).

Determining, according to the depth information and the focus area, the first depth information of the foreground area in the main image and the second depth information of the background area, and dividing the background area into different sub-areas according to the size of the second depth information, wherein the distance from the focus area is larger The smaller the depth of field of the near sub-area, the average depth of field information of different sub-areas in the background area of the main image is obtained according to the second depth information, and then the original blur of different sub-areas is obtained according to the first depth information and the average depth information of different sub-areas. strength.

Specifically, in the present example, the background area is divided into a plurality of different sub-areas according to the size of the depth of field, and further, the depth of field closest to the focus area in each sub-area and the farthest distance from the focus area are respectively obtained. The depth of field of the position is averaged according to the depth of field of the nearest position and the depth of field of the farthest position, and the average depth of field is taken as the average depth of field information of the corresponding sub-area, and then the original blurring intensity is determined according to the average depth information of the sub-area Among them, the higher the average depth of field information, the greater the original blurring intensity.

Among them, it should be emphasized that if the depth of field of the sub-area from the closest position of the focus area and the farthest distance from the focus area are not obtained due to the limitation of the depth of field calculation accuracy, the sub-area may be acquired according to the sub-area. The depth of field probability distribution in the depth of field information calculates the average depth of field of the sub-area, or the depth of field information of one or more sub-areas that are far from the focus area is inaccurate, and the focus area can be obtained according to the more accurate distance obtained. The trend of the average depth of field information of the plurality of sub-areas, and the average depth of field information of one or more sub-areas that are farther from the focus area is derived.

Further, in the embodiment of the present application, the blurring weight of each sub-area can be calculated in the same manner as the above-described example to obtain the target blurring intensity of each sub-area for blurring processing.

In summary, the background blur processing method of the present application acquires the main image captured by the main camera and the sub image captured by the sub camera, and acquires the depth information of the main image according to the main image and the sub image, and determines according to the depth information and the focus area. The original blurring intensity of different sub-regions in the background area of the main image, determining the distribution orientation of different sub-regions according to the display manner of the main image, and determining the blurring weights of different sub-regions according to the weight setting strategy corresponding to the distributed orientation, and further According to the original blurring intensity and the corresponding blurring weight of different sub-regions, the target blurring intensity of different sub-regions is determined. Finally, the main image background region is blurred according to the target blurring intensity of different sub-regions. Thereby, the effect of the blurring effect is more natural and closer to the real optical virtual focus. In order to implement the above embodiments, the present application also provides a background blur processing apparatus, and FIG. 7 is a schematic structural diagram of a background blur processing apparatus according to an embodiment of the present application. As shown in FIG. 7, the background blur processing apparatus is provided. The first obtaining module 100, the first determining module 200, the second determining module 300, the third determining module 400, and the processing module 500 are included.

The first obtaining module 100 is configured to acquire a main image captured by the main camera and a sub image captured by the sub camera, and acquire depth information of the main image according to the main image and the sub image.

The first determining module 200 is configured to determine an original blurring intensity of different sub-regions in the background area of the main image according to the depth of field information and the focus area.

In an embodiment of the present application, as shown in FIG. 8, on the basis of that shown in FIG. 7, the first determining module 200 includes a first determining unit 210, a first obtaining unit 220, and a second obtaining unit 230.

The first determining unit 210 is configured to determine, according to the depth information and the focus area, first depth information of the foreground area in the main image and second depth information of the background area in the background area of the main image.

The first obtaining unit 220 is configured to acquire, according to the second depth information, average depth information of different sub-regions in the background area of the main image.

The second obtaining unit 230 is configured to acquire the original blurring intensity of different sub-regions according to the first depth information and the average depth information of different sub-regions.

The second determining module 300 is configured to determine a distributed orientation of different sub-regions according to a display manner of the main image, and determine a blurring weight of the different sub-regions according to a weight setting policy corresponding to the distributed orientation.

In an embodiment of the present application, as shown in FIG. 9, on the basis of that shown in FIG. 7, the second determining module 300 includes a third obtaining unit 310 and a second determining unit 320.

The third obtaining unit 310 is configured to acquire positioning coordinates of different sub-regions.

The second determining unit 320 is configured to determine the blurring weights of different sub-areas according to the preset linear weight distribution curve or the nonlinear weight distribution curve according to the positioning coordinate query.

The third determining module 400 is configured to determine target blurring strengths of different sub-regions according to original blurring strengths of different sub-regions and corresponding blurring weights.

The processing module 500 is configured to perform a blurring process on the background area of the main image according to the target blurring intensity of different sub-regions.

It should be noted that the foregoing description of the method embodiment is also applicable to the device in the embodiment of the present application, and the implementation principle is similar, and details are not described herein again.

The division of each module in the background blur processing device is for illustrative purposes only. In other embodiments, the background blur processing device may be divided into different modules as needed to complete all or part of the background blur processing device. Features.

In summary, the background blur processing device of the present application acquires the main image captured by the main camera and the sub image captured by the sub camera, and acquires the depth information of the main image according to the main image and the sub image, and determines according to the depth information and the focus area. The original blurring intensity of different sub-regions in the background area of the main image, determining the distribution orientation of different sub-regions according to the display manner of the main image, and determining the blurring weights of different sub-regions according to the weight setting strategy corresponding to the distribution orientation, and further, according to The original blurring intensity and corresponding blurring weights of different sub-areas determine the target blurring intensity of different sub-areas. Finally, the main image background area is blurred according to the target blurring intensity of different sub-areas. Thereby, the effect of the blurring effect is more natural and closer to the real optical virtual focus.

In order to implement the above embodiments, the present application further provides a computer device, wherein the computer device is any device including a memory including a storage computer program and a processor running the computer program, for example, a smart phone, a personal computer, or the like. The computer device further includes an image processing circuit, and the image processing circuit may be implemented by using hardware and/or software components, and may include various processing units defining an ISP (Image Signal Processing) pipeline. Figure 10 is a schematic illustration of an image processing circuit in one embodiment. As shown in FIG. 10, for convenience of explanation, only various aspects of the image processing technique related to the embodiment of the present application are shown.

As shown in FIG. 10, the image processing circuit includes an ISP processor 1040 and a control logic 1050. The image data captured by imaging device 1010 is first processed by ISP processor 1040, which analyzes the image data to capture image statistical information that may be used to determine and/or control one or more control parameters of imaging device 1010. The imaging device 1010 (camera) may include a camera having one or more lenses 1012 and an image sensor 1014, wherein the imaging device 1010 includes two sets of cameras for implementing the background blurring method of the present application, wherein, with continued reference to FIG. Imaging device 1010 can simultaneously capture scene images based on a primary camera and a secondary camera, image sensor 1014 can include a color filter array (eg, a Bayer filter), and image sensor 1014 can acquire light intensity captured by each imaging pixel of image sensor 1014 and The wavelength information is provided and a set of raw image data that can be processed by the ISP processor 1040 is provided. Sensor 1020 can provide raw image data to ISP processor 1040 based on sensor 1020 interface type, wherein ISP processor 1040 can generate raw image data acquired by image sensor 1014 in the main camera and image sensor in the secondary camera based on sensor 1020 The original image data acquired by 1014 calculates depth information and the like. The sensor 1020 interface may utilize a SMIA (Standard Mobile Imaging Architecture) interface, other serial or parallel camera interfaces, or a combination of the above.

The ISP processor 1040 processes the raw image data pixel by pixel in a variety of formats. For example, each image pixel can have a bit depth of 8, 10, 12, or 14 bits, and the ISP processor 1040 can perform one or more image processing operations on the raw image data, collecting statistical information about the image data. Among them, image processing operations can be performed with the same or different bit depth precision.

ISP processor 1040 can also receive pixel data from image memory 1030. For example, raw pixel data is sent from the sensor 1020 interface to the image memory 1030, and the raw pixel data in the image memory 1030 is then provided to the ISP processor 1040 for processing. Image memory 1030 can be part of a memory device, a storage device, or a separate dedicated memory within an electronic device, and can include DMA (Direct Memory Access) features.

When receiving raw image data from sensor 1020 interface or from image memory 1030, ISP processor 1040 can perform one or more image processing operations, such as time domain filtering. The processed image data can be sent to image memory 1030 for additional processing prior to being displayed. The ISP processor 1040 receives the processed data from the image memory 1030 and performs image data processing in the original domain and in the RGB and YCbCr color spaces. The processed image data may be output to display 1070 for viewing by a user and/or further processed by a graphics engine or a GPU (Graphics Processing Unit). Additionally, the output of ISP processor 1040 can also be sent to image memory 1030, and display 1070 can read image data from image memory 1030. In one embodiment, image memory 1030 can be configured to implement one or more frame buffers. Additionally, the output of ISP processor 1040 can be sent to encoder/decoder 1060 to encode/decode image data. The encoded image data can be saved and decompressed before being displayed on the display 1070 device. Encoder/decoder 1060 can be implemented by a CPU or GPU or coprocessor.

The statistics determined by the ISP processor 1040 can be sent to the control logic 1050 unit. For example, the statistical data may include image sensor 1014 statistical information such as auto exposure, auto white balance, auto focus, flicker detection, black level compensation, lens 1012 shading correction, and the like. Control logic 1050 can include a processor and/or a microcontroller that executes one or more routines, such as firmware, and one or more routines can determine control parameters and control of imaging device 1010 based on received statistical data. parameter. For example, the control parameters may include sensor 1020 control parameters (eg, gain, integration time for exposure control), camera flash control parameters, lens 1012 control parameters (eg, focus or zoom focal length), or a combination of these parameters. The ISP control parameters may include gain levels and color correction matrices for automatic white balance and color adjustment (eg, during RGB processing), as well as lens 1012 shading correction parameters.

The following are the steps for implementing the background blurring method using the image processing technique in FIG. 10:

Acquiring a main image captured by the main camera and a sub image captured by the sub camera, and acquiring depth information of the main image according to the main image and the sub image;

Determining an original blurring intensity of different sub-regions in the background area of the main image according to the depth of field information and the focus area;

Determining a distribution orientation of the different sub-regions according to a display manner of the main image, and determining a blur weight of the different sub-regions according to a weight setting policy corresponding to the distribution orientation;

Determining a target blurring intensity of the different sub-regions according to an original blurring intensity of the different sub-regions and a corresponding blurring weight;

The main image background area is blurred according to the target blurring intensity of the different sub-areas.

In order to implement the above embodiments, the present application also proposes a non-transitory computer readable storage medium that enables execution of a background blurring processing method as in the above embodiment when instructions in the storage medium are executed by a processor.

In the description of the present specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means a specific feature described in connection with the embodiment or example. A structure, material or feature is included in at least one embodiment or example of the application. In the present specification, the schematic representation of the above terms is not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. In addition, various embodiments or examples described in the specification, as well as features of various embodiments or examples, may be combined and combined.

Moreover, the terms "first" and "second" are used for descriptive purposes only and are not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, features defining "first" or "second" may include at least one of the features, either explicitly or implicitly. In the description of the present application, the meaning of "a plurality" is at least two, such as two, three, etc., unless specifically defined otherwise.

Any process or method description in the flowcharts or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logic function or process. And the scope of the preferred embodiments of the present application includes additional implementations, in which the functions may be performed in a substantially simultaneous manner or in the reverse order depending on the functions involved, in accordance with the illustrated or discussed order. It will be understood by those skilled in the art to which the embodiments of the present application pertain.

The logic and/or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be embodied in any computer readable medium, Used in conjunction with, or in conjunction with, an instruction execution system, apparatus, or device (eg, a computer-based system, a system including a processor, or other system that can fetch instructions and execute instructions from an instruction execution system, apparatus, or device) Or use with equipment. For the purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport a program for use in an instruction execution system, apparatus, or device, or in conjunction with the instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer readable media include the following: electrical connections (electronic devices) having one or more wires, portable computer disk cartridges (magnetic devices), random access memory (RAM), Read only memory (ROM), erasable editable read only memory (EPROM or flash memory), fiber optic devices, and portable compact disk read only memory (CDROM). In addition, the computer readable medium may even be a paper or other suitable medium on which the program can be printed, as it may be optically scanned, for example by paper or other medium, followed by editing, interpretation or, if appropriate, other suitable The method is processed to obtain the program electronically and then stored in computer memory.

It should be understood that portions of the application can be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware and in another embodiment, it can be implemented by any one or combination of the following techniques well known in the art: discrete with logic gates for implementing logic functions on data signals Logic circuits, application specific integrated circuits with suitable combinational logic gates, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), and the like.

One of ordinary skill in the art can understand that all or part of the steps carried by the method of implementing the above embodiments can be completed by a program to instruct related hardware, and the program can be stored in a computer readable storage medium. When executed, one or a combination of the steps of the method embodiments is included.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. The integrated modules, if implemented in the form of software functional modules and sold or used as stand-alone products, may also be stored in a computer readable storage medium.

The above mentioned storage medium may be a read only memory, a magnetic disk or an optical disk or the like. While the embodiments of the present application have been shown and described above, it is understood that the above-described embodiments are illustrative and are not to be construed as limiting the scope of the present application. The embodiments are subject to variations, modifications, substitutions and variations.

Claims

A background blur processing method, comprising:

Acquiring a main image captured by the main camera and a sub image captured by the sub camera, and acquiring depth information of the main image according to the main image and the sub image;

Determining an original blurring intensity of different sub-regions in the background area of the main image according to the depth of field information and the focus area;

Determining a distribution orientation of the different sub-regions according to a display manner of the main image, and determining a blur weight of the different sub-regions according to a weight setting policy corresponding to the distribution orientation;

Determining a target blurring intensity of the different sub-regions according to an original blurring intensity of the different sub-regions and a corresponding blurring weight;

The main image background area is blurred according to the target blurring intensity of the different sub-areas.
The method according to claim 1, wherein the determining the original blurring intensity of different sub-regions in the background area of the main image according to the depth of field information and the in-focus area comprises:

Determining, according to the depth information and the focus area, first depth information of the foreground area in the main image and second depth information of the background area;

Acquiring average depth information of different sub-regions in the background area of the main image according to the second depth information;

And acquiring an original blurring intensity of the different sub-regions according to the first depth information and the average depth information of the different sub-regions.
The method according to claim 1 or 2, wherein when the distribution orientation of the different sub-regions is an up-and-down orientation, the weight setting strategy corresponding to the distribution orientation determines the imaginary of the different sub-regions The weights include:

The blurring weights of the different sub-areas are determined according to a top-down weight decrementing strategy.
The method according to claim 3, wherein the determining the blurring weights of the different sub-areas according to the top-to-bottom weight reduction strategy comprises:

Obtaining positioning coordinates of the different sub-areas;

Determining the blurring weights of the different sub-regions according to the preset linear weight distribution curve or the nonlinear weight distribution curve according to the positioning coordinate query.
The method according to any one of claims 1 to 4, wherein the determining the target blurring strength of the different sub-regions according to the original blurring intensity of the different sub-regions and the corresponding blurring weights comprises:

Obtaining the product of the original blurring intensity of the different sub-regions and the corresponding blurring weights, and determining the target blurring strength of the different sub-regions.
The method according to any one of claims 1-5, wherein the blurring of the background area of the main image according to the target blurring intensity of the different sub-regions comprises:

Determining a blurring coefficient of each pixel in the background region according to target blurring intensity of the different sub-regions and depth of field information of each pixel in the different sub-regions;

The background region is subjected to Gaussian blur processing according to a blurring coefficient of each pixel in the background region to generate a blurred photograph.
A background blur processing device, comprising:

a first acquiring module, configured to acquire a main image captured by the main camera and a sub image captured by the sub camera, and acquire depth information of the main image according to the main image and the sub image;

a first determining module, configured to determine an original blurring intensity of different sub-regions in the background area of the main image according to the depth of field information and the focus area;

a second determining module, configured to determine a distribution orientation of the different sub-regions according to a display manner of the main image, and determine a blur weight of the different sub-regions according to a weight setting policy corresponding to the distributed orientation;

a third determining module, configured to determine target blurring strengths of the different sub-regions according to original blurring strengths of the different sub-regions and corresponding blurring weights;

And a processing module, configured to perform a blurring process on the background area of the main image according to the target blurring intensity of the different sub-regions.
The apparatus of claim 7, wherein the first determining module comprises:

a first determining unit, configured to determine, according to the depth information and the in-focus area, first depth of field information of the foreground area and the second depth of field information of the background area in the main image background area;

a first acquiring unit, configured to acquire, according to the second depth information, average depth of field information of different sub-regions in the background area of the main image;

a second acquiring unit, configured to acquire an original blurring intensity of the different sub-regions according to the first depth information and the average depth information of the different sub-regions.
The apparatus according to claim 7 or 8, wherein when the distribution orientation of the different sub-areas is an up-and-down orientation, the second determining module is specifically configured to:

The blurring weights of the different sub-areas are determined according to a top-down weight decrementing strategy.
The device according to claim 9, wherein the second determining module is specifically configured to:

Obtaining positioning coordinates of the different sub-areas;

Determining the blurring weights of the different sub-regions according to the preset linear weight distribution curve or the nonlinear weight distribution curve according to the positioning coordinate query.
The device according to any one of claims 7 to 10, wherein the third determining module is specifically configured to:

Obtaining the product of the original blurring intensity of the different sub-regions and the corresponding blurring weights, and determining the target blurring strength of the different sub-regions.
The device according to any one of claims 7-11, wherein the processing module is specifically configured to:

Determining a blurring coefficient of each pixel in the background region according to target blurring intensity of the different sub-regions and depth of field information of each pixel in the different sub-regions;

The background region is subjected to Gaussian blur processing according to a blurring coefficient of each pixel in the background region to generate a blurred photograph.
A computer device, comprising: a memory, a processor, and a computer program stored on the memory and operable on the processor, wherein when the processor executes the program, implementing any one of claims 1-6 The background blurring processing method.
A computer readable storage medium having stored thereon a computer program, wherein the program is executed by a processor to implement the background blurring processing method according to any one of claims 1-6.
An image processing circuit, comprising: an ISP processor, wherein the ISP processor comprises: a main camera and a sub camera with an image sensor,

The ISP processor is configured to acquire, by using an interface of the image sensor, a main image captured by a main camera and a sub image captured by a sub camera, and acquire depth information of the main image according to the main image and the sub image. Determining an original blurring intensity of different sub-regions in the background area of the main image according to the depth information and the in-focus area, determining a distribution orientation of the different sub-areas according to a display manner of the main image, and according to the distribution orientation Determining a weighting weight of the different sub-regions, determining a target blurring strength of the different sub-regions according to the original blurring strength of the different sub-regions and the corresponding blurring weights, according to the different sub-regions The target blurring intensity of the region blurs the background area of the main image.