WO2019105260A1

WO2019105260A1 - Depth of field obtaining method, apparatus and device

Info

Publication number: WO2019105260A1
Application number: PCT/CN2018/116474
Authority: WO
Inventors: 欧阳丹; 谭国辉
Original assignee: Oppo广东移动通信有限公司
Priority date: 2017-11-30
Filing date: 2018-11-20
Publication date: 2019-06-06
Also published as: CN108053438B; CN108053438A

Abstract

The present invention provides a depth of field obtaining method, apparatus and device. The method comprises: obtaining multiple main image frames captured by a main camera and multiple auxiliary image frames captured by an auxiliary camera, and obtaining a main reference image having the highest sharpness according to the sharpness of each main image frame and each auxiliary image frame; comparing the sharpness of main images other than the main reference image and the sharpness of each auxiliary image frame with the sharpness of the main reference image to detect whether candidate main images and candidate auxiliary images that satisfy a preset screening threshold exist; if yes, obtaining image information of the main reference image, each candidate main image frame, and each candidate auxiliary image frame to determine a first target main image and a first target auxiliary image; and obtaining depth of field information according to the first target main image and the first target auxiliary image. Hence, the quality and consistency of images from which depth of field information is obtained are ensured, and the precision of the depth of field and imaging effect are improved.

Description

Depth of field acquisition method, device and device

Cross-reference to related applications

This application claims the priority of the Chinese patent application number "201711243742.7" filed on November 30, 2017 by Guangdong Opal Mobile Communications Co., Ltd., which is entitled "Deep Depth Acquisition Method, Apparatus and Equipment".

Technical field

The present application relates to the field of image processing technologies, and in particular, to a method, device, and device for acquiring depth of field.

Background technique

At present, terminal devices such as smart phones widely use a dual camera system to calculate the depth of field by two images acquired simultaneously by two cameras, for example, by the difference in position of pixels in the same position in the two images in the image. , calculate the depth of field information of the captured scene.

In the related art, the depth of field information is directly calculated according to the two images simultaneously captured by the two cameras. When the difference between the two images of the calculated depth of field is large, the two images are in the same position in the scene for shooting. Pixels are less, etc., resulting in low depth of field calculation accuracy.

Application content

The present application provides a method, a device and a device for acquiring a depth of field, so as to solve the technical problem that the two images of the image depth information are calculated to be inaccurate due to a large gap in the prior art.

The embodiment of the present application provides a depth of field acquisition method, including: acquiring a multi-frame main image captured by a main camera and a multi-frame sub-image captured by a sub-camera, and obtaining the highest resolution according to the definition of the main image and each sub-picture of each frame. a reference main image; comparing the sharpness of the remaining main images other than the reference main image and the sharpness of each sub-image with the sharpness of the reference main image, detecting whether there is a preset screening threshold a candidate primary image and a candidate secondary image; if it is detected that at least one frame of the candidate primary image and at least one frame of the candidate secondary image are present, acquiring the reference primary image, each frame candidate primary image, and each frame candidate secondary image Image information, determining a first target main image and a first target sub image; and acquiring depth information according to the first target main image and the first target sub image.

Another embodiment of the present invention provides a depth of field acquisition apparatus, including: a first acquisition module, configured to acquire a multi-frame main image captured by a main camera and a multi-frame sub-image captured by a sub-camera, and a second acquisition module, configured to a frame main image and a sharpness of each sub-picture, obtaining a reference main image with the highest definition; a detecting module for setting the sharpness of the remaining main image except the reference main image and the sharpness of each sub-picture Comparing with the definition of the reference main image, detecting whether there is a candidate main image and a candidate sub-image satisfying a preset screening threshold; and a third obtaining module, configured to detect that at least one frame of the candidate main image exists and at least And acquiring the reference main image, the candidate main image of each frame, and the image information of each candidate sub-image, determining the first target main image and the first target sub-image; and the fourth acquiring module, Depth information is obtained according to the first target main image and the first target sub image.

A further embodiment of the present application provides an acquisition device, including a memory and a processor, wherein the memory stores an acquirer readable instruction, and when the instruction is executed by the processor, the processor executes the application. The depth of field acquisition method described in the above embodiment.

A further embodiment of the present application provides a non-transitory machine readable storage medium, on which an acquirer program is stored, and when the processor program is executed by the processor, the depth of field acquisition method according to the above embodiment of the present application is implemented.

The technical solutions provided by the embodiments of the present application may include the following beneficial effects:

Acquiring the multi-frame main image captured by the main camera and the multi-frame sub-image captured by the sub-camera, calculating the definition of the main image and each sub-picture of each frame, obtaining the reference main image with the highest definition, and the clarity of the remaining main images and Comparing the sharpness of each sub-picture with the sharpness of the reference main image, detecting whether there is a candidate main image and a candidate sub-image satisfying the preset screening threshold, if the detection is that at least one frame candidate main image and at least one frame candidate pair are present And acquiring image information of the reference main image, each candidate main image, and each candidate sub-image, determining the first target main image and the first target sub image, and further, according to the first target main image and the first target sub image Get depth of field information. Thereby, the quality and consistency between the images for acquiring the depth information are ensured, and the accuracy of the depth of field and the imaging effect are improved.

DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings used in the embodiments will be briefly described below. Obviously, the drawings in the following description are some embodiments of the present application. Those skilled in the art can also obtain other drawings based on these drawings without paying any creative work.

1 is a schematic diagram of a principle of triangulation according to an embodiment of the present application;

2 is a schematic diagram of a process of calculating a depth of field by a dual camera according to an embodiment of the present application;

3 is a flowchart of a depth of field acquisition method according to an embodiment of the present application;

4(a) is a schematic diagram of a scene of a depth of field acquisition method according to an embodiment of the present application;

4(b) is a schematic diagram of a scene of a depth of field acquisition method according to another embodiment of the present application;

FIG. 5 is a schematic structural diagram of a depth of field acquiring apparatus according to an embodiment of the present application;

6 is a schematic diagram of an image processing circuit in accordance with one embodiment of the present application.

Detailed ways

The embodiments of the present application are described in detail below, and the examples of the embodiments are illustrated in the drawings, wherein the same or similar reference numerals are used to refer to the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are intended to be illustrative, and are not to be construed as limiting.

The method, device and device for acquiring the depth of field according to the embodiment of the present application are described below with reference to the accompanying drawings. The depth of field acquisition method in the embodiment of the present application is applicable to a hardware device having a dual camera, such as a mobile phone, a tablet computer, a personal digital assistant, a wearable device, and the like, and the wearable device may be a smart bracelet, a smart watch, smart glasses, or the like.

It should be understood that the dual camera system calculates the depth of field through the main image and the sub image. In order to more clearly describe how the dual camera acquires the depth of field information, the principle of acquiring the depth of field by the dual camera will be described below with reference to the accompanying drawings:

In practical applications, the human eye distinguishes the depth of field mainly by relying on binocular vision to distinguish the depth of field. This is the same as the principle of dual camera resolution depth of field, mainly based on the principle of triangular ranging as shown in Figure 1, based on Figure 1, In the actual space, the imaged object is drawn, as well as the positions of the two cameras O _R and O _T , and the focal planes of the two cameras. The focal plane is at a distance f from the plane of the two cameras, and two at the focal plane. The camera performs imaging to obtain two captured images.

Among them, P and P' are the positions of the same object in different captured images, respectively. The distance from the P point to the left boundary of the captured image is X _R , and the distance from the P′ point to the left boundary of the captured image is X _T . O _R and O _T are two cameras respectively, and the two cameras are on the same plane with a distance B.

Based on the principle of triangulation, the distance Z between the object in Figure 1 and the plane of the two cameras has the following relationship:

Based on this, you can push

Where d is the difference in distance between the positions of the same object in different captured images. Since B and f are constant values, the distance Z of the object can be determined according to d.

Of course, in addition to the triangulation method, other methods can be used to calculate the depth of field of the main image. For example, when the main camera and the sub camera are photographed for the same scene, the distance between the object in the scene and the camera is imaged by the main camera and the sub camera. The displacement difference, the posture difference, and the like are proportional, and therefore, in one embodiment of the present application, the above-described distance Z can be obtained according to such a proportional relationship.

For example, as shown in FIG. 2, a map of different point differences is calculated by the main image acquired by the main camera and the sub-image obtained by the sub-camera, which is represented by a disparity map, which is the same on the two graphs. The difference in displacement of the points, but since the difference in displacement in the triangulation is proportional to Z, many times the parallax chart is directly used as the depth of field map.

Based on the above analysis, when the dual camera acquires the depth of field, it is necessary to acquire the position of the same object in different captured images. Therefore, if the two images of the dual cameras that acquire the depth information are relatively close, the efficiency and accuracy of the depth of field acquisition are improved.

FIG. 3 is a flowchart of a method for acquiring a depth of field according to an embodiment of the present application. As shown in FIG. 3, the method includes:

Step 101: Acquire a multi-frame main image captured by the main camera and a multi-frame sub-image captured by the sub-camera.

Step 102: Acquire a reference main image with the highest definition according to the definition of the main image and the sub-image of each frame.

Among them, the sharpness of the image refers to the degree of clarity of the edge of the image, including the distinction between the lines of the image, that is, the resolution of the image point or the fineness of the texture of the subtle layer. The higher the resolution, the material of the scene. The resolution of the point or the fineness of the texture of the subtle layer is higher. The finer the performance of the material point, the higher the definition. The definition also includes whether the contour of the edge of the line is clear, that is, the degree of the virtual boundary of the contour of the image, the common sharpness. It means that the essence is the variation width of the gradient density of the hierarchical boundary. If the variation width is small, the boundary is clear. Conversely, if the variation width is large, the boundary is faint, and the definition also includes the degree of clarity between the small levels, especially the contrast between the small layers. Or is the slight contrast clear?

That is to say, the higher the sharpness of the image, the easier it is to distinguish the edge details of the image, and the less the noise, the higher the efficiency and accuracy of the depth of field calculation according to the image.

Specifically, in the embodiment, the multi-frame main image captured by the main camera and the multi-frame sub-image captured by the sub-camera are acquired, and the definition of the main image and the sub-image of each frame is calculated, and the reference master with the highest definition is obtained. The image is used to make the reference main image as a reference, and the image with high definition is screened out as an image for further calculating the depth of field.

Step 103: Compare the sharpness of the remaining main images except the reference main image and the sharpness of each sub-image with the sharpness of the reference main image, and detect whether there are candidate main images and candidate pairs that satisfy the preset screening threshold. image.

The preset screening threshold is used to filter the main image and the sub-image with higher definition sharpness than the reference main image. For example, if the preset screening threshold is 80%, the preset screening threshold may be used to filter out An image that meets the definition of the resolution of the main image by more than 80%. Specifically, comparing the sharpness of the remaining main image and the sharpness of the sub-image with the sharpness of the reference main image, and detecting whether there is a candidate main image satisfying the preset screening threshold based on the reference main image with higher definition And the candidate sub-image to determine whether there is a main image and a sub-image with higher definition. Thereby, the determination of the candidate main image and the candidate sub-image is performed based on the sharpness of the reference image, and the photographing ability of the terminal device in the current scene is considered, and the flexibility of filtering out the candidate main image and the candidate sub-image is improved.

Step 104: If it is detected that there is at least one frame candidate main image and at least one frame candidate sub-image, calculate image information of the reference main image, each frame candidate main image, and each frame candidate sub-image, and determine the first target main image and the first A target secondary image.

Step 105: Calculate depth information according to the first target main image and the first target sub image.

Specifically, in an embodiment of the present application, if it is detected that there is at least one frame candidate main image and at least one frame candidate sub-image, it indicates that there is a main image and a sub-image with higher definition, if the definition is higher The candidate main image and the candidate sub-image calculate the depth of field, which will improve the efficiency and accuracy of depth of field calculation.

In the actual execution process, the closer the main image and the sub-image of the depth of field are calculated, the smaller the interference when calculating the depth of field, and the calculated depth of field is more accurate. Therefore, in the embodiment of the present application, the reference main image and each frame candidate are acquired. And comparing image information of the main image and the candidate sub-image of each frame, wherein the image information includes, but is not limited to, image sharpness, brightness, AWB (Automatic White Balance), etc., which affect the depth of field calculation, and further Determining a first target main image and a first target sub image, for example, acquiring a first target main image and a first target sub image whose image difference satisfies a preset condition, according to the first target main image and the first target sub image Obtaining the depth of field information, thereby ensuring that the calculation of the depth of field information is more accurate, so that the final imaging effect is better.

The preset condition is related to the specific information included in the image information and the camera hardware capability and the photographing environment of the terminal device. The richer the specific information included in the terminal device, the worse the camera hardware capability of the terminal device, and the less the light of the photographing environment, The more relaxed the preset condition is, the larger the value of the corresponding image difference is. For example, under the camera hardware capability and the photographing environment of the same terminal device, the preset condition set under the scene in which the image information includes image sharpness and brightness may be a reference. The image information difference between the main image, the candidate main image per frame, and the candidate sub-image per frame is 10%, and the preset conditions set under the scene in which the image information includes image sharpness may be a reference main image, a candidate main image per frame, and each The image information difference of the frame candidate sub-image is 15%.

For a clearer explanation, how to judge whether the difference between the image information of the reference main image, the candidate main image of each frame, and the image information of each candidate sub-image satisfies a preset condition, and the following information content includes a type of information and The types of information are described as an example:

In the first example, in this example, the acquired image information is one type of information, for example, all of the image brightness.

Specifically, in this example, the image information of the reference main image and the candidate main image of each frame is sequentially compared with the image information of each candidate sub-image, and the two images with the smallest difference of the image information are acquired as the first target main image. Image and first sub image.

For example, when the image information is brightness information, as shown in FIG. 4( a ), the main camera and the sub-camera are simultaneously photographed, and a 4-frame main image and a 4-frame sub-image are acquired, wherein the main image is 4 frames in accordance with the shooting order. The numbers are 11, 12, 13 and 14, respectively, and the number of sub-images of 4 frames is 21, 22, 23 and 24, respectively, wherein the reference main image with the highest definition is 11, and the candidate main images are 12 and 13, respectively. Images, candidate secondary images are 22 and 24, respectively. The image brightness of the reference main image and the candidate main image of each frame is sequentially compared with the image brightness of each candidate sub-image, and the two image images with the smallest image brightness difference are obtained as the first target main image 12 and the first target sub image. 22, whereby the depth of field information is calculated based on the first target main image 12 and the first target sub image 22, and the final imaging effect is better according to the depth information.

In the second example, in this example, the acquired image information is a plurality of types of information including, for example, image brightness, image white balance value, and image sharpness.

Specifically, in this example, a weighting factor corresponding to each type of information is obtained, and the weighting value corresponding to the weighting factor may be calibrated by the system, or may be calibrated by the user according to the needs of the scene, and the reference main image and each frame candidate may be referred to. Each type of image information of the main image is sequentially compared with each type of image information of each candidate sub-image, and information difference of each type of image information between each two frames of images is obtained, according to information of various types of image information between each two frames of images. The difference and the weighting factor corresponding to each type of information acquire the information difference corresponding to each two frames of images, and the two frames with the smallest information difference are the first target main image and the first target sub image.

For example, when the image information is image brightness, AWB, and SOF, and the corresponding weight factors are 50%, 20%, and 30%, respectively, as shown in FIG. 4(b), the main camera and the sub-camera are simultaneously photographed, and 4 is acquired. a frame main image and a 4-frame sub-image, wherein the numbers of the main frames of the four frames in the order of shooting are 11, 12, 13, and 14, respectively, and the numbers of the sub-images of the four frames are 21, 22, 23, and 24, respectively, The highest reference main image is 11, the candidate main images are 12 and 13, respectively, and the candidate sub-images are 22 and 24, respectively. The image brightness, AWB, and SOF of the reference main image and each candidate candidate main image are sequentially compared with the image brightness of each candidate sub-image, respectively, and image brightness, AWB, and SOF of the reference main image 11 and the candidate sub-image 22 are acquired. The information difference is a1, a2, and a3, respectively, and the information difference b1=a1*50%+a2*20%+a*30% of the reference main image 11 and the candidate sub-image 22 is acquired, and so on, respectively, and the reference main is obtained. The information difference b2 between the image 11 and the candidate sub-image 24, the information difference b3 between the candidate main image 12 and the candidate sub-image 22, the information difference b4 between the candidate main image 12 and the candidate sub-image 24, and the candidate main image 13 and the candidate sub-image 22 The information difference b5, the information difference b6 between the candidate main image 13 and the candidate sub-image 24, and further, the two frames obtained with the smallest information difference are the first target main image 12 and the first target sub-image 22, thereby, according to the first target The main image 12 and the first target sub-image 22 are more accurate in calculating the depth of field information, and the final imaging effect is better according to the depth information.

Based on the above examples, it should be noted that the specific type of image information may be determined by one or more of the captured scene information and the shooting mode. For example, if the brightness of the scene information is poor, the multi-frame main image is captured. And the sub-picture quality is poor, and determining the first target main image and the first target sub-image for calculating the depth of field based on only one type of image information may not be highly reliable, and therefore, various types of image information need to be considered to determine the first target. The main image and the first target sub-image, for example, the brightness of the light in the scene information is good, and the quality of the multi-frame main image and the sub-image captured is high, and the first target main image for calculating the depth of field is determined only according to one type of image information. The first target sub-image is highly reliable, and thus, in order to improve image processing efficiency, one type of image information may be considered to determine the first target main image and the first target sub-image.

For example, if the current shooting mode is night scene shooting, the requirement for brightness information is high, and the brightness of the light is poor, and the quality of the multi-frame main image and the sub-image taken is poor, and only the first of the calculated depth of field is determined according to one type of image information. The target main image and the first target sub image may not be highly reliable. Therefore, it is necessary to consider various types of image information to determine the first target main image and the first target sub image, and, for example, in the highlight shooting mode, The problem that is easily caused is overexposure. Therefore, in order to improve image processing efficiency, one type of AWB information can be considered to determine the first target main image and the first target sub image.

Specifically, in this example, the captured scene information is detected, and/or the shooting mode, and further, the image information type to be acquired is determined according to the scene information, and/or the shooting mode, for example, the scene information may be pre-stored. And/or, the correspondence between the shooting mode and the image information type, and further, after learning the current scene information, and/or the shooting mode, querying the corresponding relationship, and acquiring the corresponding image information type.

It should be emphasized that, in the above embodiment, the implementation manner of determining the type of the image information by using the scene information or the shooting mode alone includes the implementation of determining the type of the image information by using the scene information and the shooting mode at the same time.

In an embodiment of the present application, if the detection knows that there is no candidate main image or candidate sub-image, that is, the resolutions of the remaining main image and the image may be relatively low, and at this time, the reference main image with higher definition is Obtaining a frame image of the depth information, acquiring the reference main image and the image information of each sub image, and comparing, obtaining a second target sub image whose image information difference satisfies a preset condition, wherein the second target sub image is a reference main image The closest one sub-picture, and further, the depth information is obtained with the reference main image and the second target sub-image.

Therefore, the depth of field acquisition method of the embodiment of the present application, by taking a multi-frame main and sub-image, selects a set of pictures whose main image has good definition and high definition of the main and sub-images, and the image information is as close as possible, for calculating Depth of field and final imaging, which makes the depth of field calculation more accurate, while at the same time ensuring image clarity and better final imaging.

In summary, the depth of field acquisition method of the embodiment of the present application acquires a multi-frame main image captured by a main camera and a multi-frame sub-image captured by a sub-camera, and calculates the definition of the main image and each sub-picture of each frame to obtain the definition. The highest reference main image, comparing the sharpness of the remaining main images with the sharpness of each sub-image and the sharpness of the reference main image, detecting whether there is a candidate main image and a candidate sub-image satisfying the preset screening threshold, if the detection Obtaining that at least one frame candidate main image and at least one frame candidate sub image exist, acquiring image information of the reference main image, each frame candidate main image, and each frame candidate sub image, and determining the first target main image and the first target sub image, Further, depth information is acquired based on the first target main image and the first target sub image. Thereby, the quality and consistency between the images for acquiring the depth information are ensured, and the accuracy of the depth of field and the imaging effect are improved.

In order to implement the above embodiment, the present application also provides a depth of field acquisition device. FIG. 5 is a schematic structural diagram of a depth of field acquisition device according to an embodiment of the present application. As shown in FIG. 5, the depth of field information acquisition device includes a first acquisition module. 100. The second obtaining module 200, the detecting module 300, the third obtaining module 400, and the determining module 500 and the fourth obtaining module 600.

The first obtaining module 100 is configured to acquire a multi-frame main image captured by the main camera and a multi-frame sub-image captured by the sub-camera.

The second obtaining module 200 is configured to obtain a reference main image with the highest definition according to the definition of each frame main image and each sub-picture.

The detecting module 300 is configured to compare the sharpness of the remaining main images except the reference main image and the sharpness of each sub-image with the sharpness of the reference main image, and detect whether there is a candidate main image that satisfies the preset screening threshold. And candidate secondary images.

The third obtaining module 400 is configured to acquire image information of the reference main image, each frame candidate main image, and each frame candidate sub-image when detecting that at least one frame candidate main image and at least one frame candidate sub-image exist.

The determining module 500 is configured to determine the first target primary image and the first target secondary image.

The fourth obtaining module 600 is configured to obtain depth information according to the first target main image and the first target sub image.

In an embodiment of the present application, when the acquired image information is one type of information, the second obtaining module 200 is specifically configured to sequentially compare the image information of the reference main image and the candidate main image with each frame candidate The image information of the image is compared, and two frames of images having the smallest difference in image information are acquired as the first target main image and the first target sub image.

In an embodiment of the present application, the third obtaining module 400 is further configured to: satisfy a difference between the image information in the multi-frame sub-image and the image information of the reference main image when detecting that the candidate main image or the candidate sub-image is not present. The sub-picture of the preset condition is used as the second target sub-picture.

The fourth obtaining module 600 is further configured to obtain depth information according to the reference main image and the second target sub image.

It should be noted that the foregoing description of the method embodiment is also applicable to the device in the embodiment of the present application, and the implementation principle is similar, and details are not described herein again.

The division of each module in the above-mentioned depth of field acquisition device is for illustrative purposes only. In other embodiments, the depth of field acquisition device may be divided into different modules as needed to complete all or part of the functions of the depth of field acquisition device.

In summary, the depth of field acquisition device of the embodiment of the present application acquires a multi-frame main image captured by a main camera and a multi-frame sub-image captured by a sub-camera, and calculates the definition of the main image and each sub-picture of each frame to obtain the definition. The highest reference main image, comparing the sharpness of the remaining main images with the sharpness of each sub-image and the sharpness of the reference main image, detecting whether there is a candidate main image and a candidate sub-image satisfying the preset screening threshold, if the detection Obtaining that at least one frame candidate primary image and at least one frame candidate secondary image are present, then acquiring the reference primary image, each frame candidate primary image, and the image information of each frame candidate secondary image to determine the first target primary image and the first target secondary image, and further And acquiring depth information according to the first target main image and the first target sub image. Thereby, the quality and consistency between the images for acquiring the depth information are ensured, and the accuracy of the depth of field and the imaging effect are improved.

In order to implement the above embodiments, the present application further provides a computer device, wherein the computer device is any device including a memory including a storage computer program and a processor running the computer program, for example, a smart phone, a personal computer, or the like. The above computer device includes an image processing circuit, and the image processing circuit may be implemented by hardware and/or software components, and may include various processing units defining an ISP (Image Signal Processing) pipeline. Figure 6 is a schematic illustration of an image processing circuit in one embodiment. As shown in FIG. 6, for convenience of explanation, only various aspects of the image processing technique related to the embodiment of the present application are shown.

As shown in FIG. 6, the image processing circuit includes an ISP processor 640 and a control logic 650. The image data captured by imaging device 610 is first processed by ISP processor 640, which analyzes the image data to capture image statistical information that can be used to determine and/or control one or more control parameters of imaging device 610. The imaging device 610 (camera) may include a camera having one or more lenses 612 and an image sensor 614, wherein the imaging device 610 includes two sets of cameras for implementing the background blurring method of the present application, wherein, with continued reference to FIG. 6, Imaging device 610 can simultaneously capture scene images based on a primary camera and a secondary camera, image sensor 614 can include a color filter array (such as a Bayer filter), and image sensor 614 can acquire light intensity captured by each imaging pixel of image sensor 614 and Wavelength information and a set of raw image data that can be processed by ISP processor 640. Sensor 620 can provide raw image data to ISP processor 640 based on sensor 620 interface type, wherein ISP processor 640 can be based on raw image data acquired by image sensor 614 in the main camera provided by sensor 620 and image sensor in the secondary camera The original image data acquired by 614 calculates depth information and the like. The sensor 620 interface may utilize a SMIA (Standard Mobile Imaging Architecture) interface, other serial or parallel camera interfaces, or a combination of the above.

The ISP processor 640 processes the raw image data pixel by pixel in a variety of formats. For example, each image pixel can have a bit depth of 8, 10, 12, or 14 bits, and the ISP processor 640 can perform one or more image processing operations on the raw image data, collecting statistical information about the image data. Among them, image processing operations can be performed with the same or different bit depth precision.

ISP processor 640 can also receive pixel data from image memory 630. For example, raw pixel data is sent from the sensor 620 interface to image memory 630, which is then provided to ISP processor 640 for processing. Image memory 630 can be part of a memory device, a storage device, or a separate dedicated memory within an electronic device, and can include DMA (Direct Memory Access) features.

When receiving raw image data from the sensor 620 interface or from image memory 630, ISP processor 640 can perform one or more image processing operations, such as time domain filtering. The processed image data can be sent to image memory 630 for additional processing before being displayed. The ISP processor 640 receives the processed data from the image memory 630 and performs image data processing in the original domain and in the RGB and YCbCr color spaces. The processed image data can be output to display 670 for viewing by a user and/or further processed by a graphics engine or GPU (Graphics Processing Unit). Additionally, the output of ISP processor 640 can also be sent to image memory 630, and display 670 can read image data from image memory 630. In one embodiment, image memory 630 can be configured to implement one or more frame buffers. Additionally, the output of ISP processor 640 can be sent to encoder/decoder 660 to encode/decode image data. The encoded image data can be saved and decompressed before being displayed on the display 670 device. Encoder/decoder 660 can be implemented by a CPU or GPU or coprocessor.

The statistics determined by the ISP processor 640 can be sent to the control logic 650 unit. For example, the statistics may include image sensor 614 statistics such as auto exposure, auto white balance, auto focus, flicker detection, black level compensation, lens 612 shading correction, and the like. Control logic 650 can include a processor and/or a microcontroller that executes one or more routines (such as firmware) that can determine control parameters and control of imaging device 610 based on received statistical data. parameter. For example, the control parameters may include sensor 620 control parameters (eg, gain, integration time for exposure control), camera flash control parameters, lens 612 control parameters (eg, focus or zoom focal length), or a combination of these parameters. The ISP control parameters may include gain levels and color correction matrices for automatic white balance and color adjustment (eg, during RGB processing), as well as lens 612 shading correction parameters.

The following are the steps to implement the depth of field acquisition method using the image processing technique in Figure 6:

Acquiring a multi-frame main image captured by the main camera and a multi-frame sub-image captured by the sub-camera;

Obtaining the highest-resolution reference main image according to the definition of the main image of each frame and the sub-image of each frame;

Comparing the sharpness of the remaining main images other than the reference main image and the sharpness of each sub-picture with the sharpness of the reference main image, detecting whether there are candidate main images and candidates satisfying the preset screening threshold Secondary image

If it is detected that there are at least one frame of the candidate main image and at least one frame of the candidate sub-image, acquiring the reference main image, each frame candidate main image, and image information of each frame candidate sub-image, determining the first target main An image and a first target sub-image;

Depth of field information is acquired based on the first target main image and the first target sub image.

In order to implement the above embodiments, the present application also proposes a non-transitory computer readable storage medium that enables execution of the depth of field acquisition method as described in the above embodiments when instructions in the storage medium are executed by the processor.

In the description of the present specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means a specific feature described in connection with the embodiment or example. A structure, material or feature is included in at least one embodiment or example of the application. In the present specification, the schematic representation of the above terms is not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. In addition, various embodiments or examples described in the specification, as well as features of various embodiments or examples, may be combined and combined.

Moreover, the terms "first" and "second" are used for descriptive purposes only and are not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, features defining "first" or "second" may include at least one of the features, either explicitly or implicitly. In the description of the present application, the meaning of "a plurality" is at least two, such as two, three, etc., unless specifically defined otherwise.

Any process or method description in the flowcharts or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a custom logic function or process. And the scope of the preferred embodiments of the present application includes additional implementations, in which the functions may be performed in a substantially simultaneous manner or in an opposite order depending on the functions involved, in the order shown or discussed. It will be understood by those skilled in the art to which the embodiments of the present application pertain.

The logic and/or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be embodied in any computer readable medium, Used in conjunction with, or in conjunction with, an instruction execution system, apparatus, or device (eg, a computer-based system, a system including a processor, or other system that can fetch instructions and execute instructions from an instruction execution system, apparatus, or device) Or use with equipment. For the purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport a program for use in an instruction execution system, apparatus, or device, or in conjunction with the instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer readable media include the following: electrical connections (electronic devices) having one or more wires, portable computer disk cartridges (magnetic devices), random access memory (RAM), Read only memory (ROM), erasable editable read only memory (EPROM or flash memory), fiber optic devices, and portable compact disk read only memory (CDROM). In addition, the computer readable medium may even be a paper or other suitable medium on which the program can be printed, as it may be optically scanned, for example by paper or other medium, followed by editing, interpretation or, if appropriate, other suitable The method is processed to obtain the program electronically and then stored in computer memory.

It should be understood that portions of the application can be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware and in another embodiment, it can be implemented by any one or combination of the following techniques well known in the art: discrete with logic gates for implementing logic functions on data signals Logic circuits, application specific integrated circuits with suitable combinational logic gates, programmable gate arrays (PGAs), field programmable gate arrays (FPGAs), and the like.

One of ordinary skill in the art can understand that all or part of the steps carried by the method of implementing the above embodiments can be completed by a program to instruct related hardware, and the program can be stored in a computer readable storage medium. When executed, one or a combination of the steps of the method embodiments is included.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. The integrated modules, if implemented in the form of software functional modules and sold or used as stand-alone products, may also be stored in a computer readable storage medium.

The above mentioned storage medium may be a read only memory, a magnetic disk or an optical disk or the like. While the embodiments of the present application have been shown and described above, it is understood that the above-described embodiments are illustrative and are not to be construed as limiting the scope of the present application. The embodiments are subject to variations, modifications, substitutions and variations.

Claims

A depth of field acquisition method, comprising:

Acquiring a multi-frame main image captured by the main camera and a multi-frame sub-image captured by the sub-camera;

Obtaining the highest-resolution reference main image according to the definition of the main image of each frame and the sub-image of each frame;

Comparing the sharpness of the remaining main images other than the reference main image and the sharpness of each sub-picture with the sharpness of the reference main image, detecting whether there are candidate main images and candidates satisfying the preset screening threshold Secondary image

If it is detected that there are at least one frame of the candidate main image and at least one frame of the candidate sub-image, according to the reference main image, each frame candidate main image, and image information of each frame candidate sub-image,

Determining a first target main image and a first target sub image;

Depth of field information is acquired based on the first target main image and the first target sub image.
The method of claim 1, wherein the obtaining the depth of field information according to the first target main image and the first target sub image comprises:

Obtaining pixel points of the same photographic subject in the first target main image and the first target sub image;

Obtaining a difference in displacement of the pixel points;

The depth of field information is determined based on the displacement difference.
The method according to claim 1, wherein after the detecting whether there is a candidate main image and a candidate sub-image satisfying a preset screening threshold, the method further comprises:

If the detection is that the candidate main image or the candidate sub image is not present, the sub image in which the difference between the image information in the multi-frame sub-image and the image information of the reference main image satisfies a preset condition is used as the second target Secondary image

Depth of field information is obtained based on the reference main image and the second target sub image.
The method according to any one of claims 1 to 3, wherein when the acquired image information is a type information, wherein the type information includes image brightness, or image white balance value, or image resolution rate,

Acquiring and comparing the image information of the reference main image, the candidate main image, and the candidate sub-image of each frame, and acquiring the first target main image and the first target sub-image, where the image information difference meets the preset condition, including:

Comparing the reference main image and the image information of each candidate candidate main image to the image information of each candidate sub-image in sequence, and acquiring two frames of images having the smallest difference of the image information as the first target main image and the first target sub image .
The method according to any one of claims 1 to 4, wherein when the acquired image information is a plurality of types of information,

And acquiring the image information difference of the image information, and determining the first target main image and the first target sub image that meet the preset condition by the image information difference, including:

Obtaining a weighting factor corresponding to each type of information;

And comparing each type of image information of the reference main image and each candidate candidate main image to each type of image information of each candidate sub-image, and acquiring information difference of each type of image information between each two frames of images;

According to the information difference of each type of image information between each two frames of images and the weighting factor corresponding to each type of information, the information difference corresponding to each two frames of images is obtained, and the two frames with the smallest information difference are obtained as the first target main image and the first A target secondary image.
The method of any of claims 1-5, further comprising:

Detecting scene information of the shooting, and/or shooting mode;

Determining the type of the image information to be acquired according to the scene information, and/or the shooting mode.
A depth of field acquiring device, comprising:

a first acquiring module, configured to acquire a multi-frame main image captured by the main camera and a multi-frame sub-image captured by the sub-camera;

a second acquiring module, configured to obtain a reference main image with the highest definition according to the definition of the main image of each frame and the sub-image of each frame;

a detecting module, configured to compare the sharpness of the remaining main images except the reference main image and the sharpness of each sub-image with the sharpness of the reference main image, and detect whether there is a preset screening threshold a candidate primary image and a candidate secondary image;

a third acquiring module, configured to acquire an image of the reference main image, each frame candidate main image, and each candidate sub-image when detecting that at least one frame of the candidate main image and at least one frame of the candidate sub-image are detected information;

a determining module, configured to determine the first target primary image and the first target secondary image;

And a fourth acquiring module, configured to acquire depth information according to the first target main image and the first target secondary image.
The device of claim 7 wherein:

The third obtaining module is further configured to: when detecting that the candidate main image or the candidate sub-image is not present, the difference between the image information in the multi-frame sub-image and the image information of the reference main image is satisfied a secondary image of a preset condition as a second target secondary image;

The fourth acquiring module is further configured to acquire depth information according to the reference main image and the second target sub image.
The apparatus according to claim 7 or 8, wherein when the acquired image information is a type information, wherein the type information includes image brightness, or image white balance value, or image resolution,

The second acquiring module is specifically configured to:

Comparing the reference main image and the image information of each candidate candidate main image to the image information of each candidate sub-image in sequence, and acquiring two frames of images having the smallest difference of the image information as the first target main image and the first target sub image .
The device according to any one of claims 7-9, wherein when the acquired image information is a plurality of types of information, the determining module comprises:

a determining unit, configured to acquire a weighting factor corresponding to each type of information;

a first acquiring unit, configured to compare each type of image information of the reference main image and each candidate candidate main image with each type of image information of each candidate sub-image, and obtain various types of image information between each two frames of images. Poor information;

a second acquiring unit, configured to acquire, according to information difference of each type of image information between each two frames of images and a weighting factor corresponding to each type of information, information difference corresponding to each two frames of images, and obtain two frames with the smallest information difference as The first target main image and the first target sub image.
A device according to any of claims 7-10, wherein

The detecting module is further configured to detect captured scene information, and/or a shooting mode;

The determining module is further configured to determine, according to the scene information, and/or a shooting mode, a type of the image information to be acquired.
An acquisition machine device, comprising: a memory, a processor, and an acquirer program stored on the memory and operable on the processor, wherein when the processor executes the program, the implementation is as claimed in claims 1-5 Any of the depth of field acquisition methods described.
An acquisition machine readable storage medium having stored thereon an acquisition program, wherein the program is executed by a processor to implement the depth of field acquisition method according to any one of claims 1-5.
An image processing circuit, comprising: an ISP processor, wherein the ISP processor comprises: a main camera and a sub camera having an image sensor, the ISP processor, and an interface through the image sensor Acquiring the multi-frame main image captured by the main camera and the multi-frame sub-image captured by the sub-camera, and obtaining the reference main image with the highest definition according to the definition of the main image and the sub-image of each frame, Comparing the sharpness of the remaining main images other than the main image and the sharpness of each sub-image with the sharpness of the reference main image, detecting whether there is a candidate main image and a candidate sub-image satisfying the preset screening threshold, if Determining that there is at least one frame of the candidate main image and at least one frame of the candidate sub-image, determining the first target main image according to the reference main image, each frame candidate main image, and image information of each frame candidate sub-image And the first target sub image, and acquiring the depth information according to the first target main image and the first target sub image.
The image processing circuit according to claim 14, wherein the ISP processor is specifically configured to:

Obtaining pixel points of the same photographic subject in the first target main image and the first target sub image;

Obtaining a difference in displacement of the pixel points;

The depth of field information is determined based on the displacement difference.