WO2018076529A1

WO2018076529A1 - Scene depth calculation method, device and terminal

Info

Publication number: WO2018076529A1
Application number: PCT/CN2016/112696
Authority: WO
Inventors: 唐忠伟; 徐荣跃; 王运; 李邢; 李远友; 敖欢欢
Original assignee: 华为技术有限公司
Priority date: 2016-10-25
Filing date: 2016-12-28
Publication date: 2018-05-03
Also published as: CN108260360B; CN108260360A

Abstract

Embodiments of the present invention relate to a scene depth calculation method, a device and a terminal. The method comprises: obtaining a camera offset of a camera having an OIS system, wherein a first camera and/or a second camera have the OIS system; converting the camera offset into an image offset according to a preset OIS motor sensitivity calibration parameter; calculating the scene depth of a target scene according to a compensated calibration parameter of the first camera and/or a compensated calibration parameter of the second camera, as well as a first image and a second image obtained by the first camera and the second camera photographing the target scene at the same time; wherein the calibration parameter of the first camera is compensated according to the camera offset of the first camera, and the calibration parameter of the second camera is compensated according to the camera offset of the second camera. Therefore, the parallax problem is solved, and the scene depth of a target scene is more accurate as being calculated using the compensated calibration parameters of cameras.

Description

Scene depth calculation method, device and terminal

This application claims priority on the Chinese Patent Application filed on October 25, 2016, the Patent Office of the State Intellectual Property Office of China, Application No. 201610941102.2, and the invention titled "Method and Terminal for Image Depth Calculation with Dual Camera Terminals" The entire content of which is incorporated herein by reference.

Technical field

The embodiments of the present invention relate to the field of communications, and in particular, to a method, a device, and a terminal for calculating a scene depth of a target scene having a dual camera terminal device.

Background technique

Currently, the method of deep calculation through dual cameras is being applied to more and more mobile phones, such as the Huawei P9 mobile phone. Optical Image Stabilization (OIS, also commonly referred to as optical image stabilization) is an important means of improving the quality of photographs in low light, and is also used on more and more mobile phones. OIS works by compensating for hand shake by moving the lens to achieve image stabilization.

In the prior art, the image is generally corrected by the calibration parameters of the dual camera, so that the left and right images provided by the dual camera are aligned in one direction, then the parallax is calculated, and the parallax is converted into the depth of the scene. However, as long as one of the cameras has an OIS system, the OIS causes a lens shift, which causes the dual camera calibration parameters to change, resulting in parallax problems (positive parallax and negative parallax exist simultaneously, or the image cannot be aligned in one direction). The calculated scene depth value is not accurate.

Summary of the invention

The embodiment of the invention provides a scene depth calculation method. When the OIS system exists in one or two cameras, the scene of the target scene caused by the parallax problem (the positive or negative parallax exists simultaneously or the image cannot be aligned in one direction) is solved. The problem of inaccurate depth values.

In a first aspect, a scene depth calculation method is provided, the method comprising: acquiring a lens offset of a camera with an OIS system; wherein the first camera and/or the second camera has an OIS system, the first camera And the second camera is arranged side by side on the body of the same terminal device; according to the preset OIS motor sensitivity calibration parameter, the lens offset is converted into an image offset; according to the compensated first camera calibration parameter and / Or the compensated second camera calibration parameter, and the first image and the second image obtained by the first camera and the second camera respectively acquiring the target scene at the same time, and calculating the scene depth of the target scene; The calibration parameter of the first camera is compensated according to the lens offset of the first camera, and the calibration parameter of the second camera is compensated according to the lens offset of the second camera. By compensating for the change of the camera calibration parameters caused by the jitter of the terminal device, the parallax problem is solved, and the scene depth of the target scene is calculated by using the compensated camera calibration parameters, and the calculated scene depth value is more accurate.

With reference to the first aspect, in a first possible implementation manner of the first aspect, the method may further include: acquiring angular velocity information of the terminal device jitter detected by the gyro sensor; converting the angular velocity information into jitter of the terminal device The amplitude drives the OIS motor to push the lens of the first camera and/or the lens of the second camera according to the amplitude of the shake, and acquires a lens offset of the first camera and/or the second camera. Wherein, because the terminal device jitter time and the exposure time are inconsistent, and the jitter time is greater than the exposure time, multiple lens offsets are acquired, and one lens offset is determined from the plurality of lens offsets according to a preset rule. And when the scene depth of the target scene is subsequently calculated, the determined lens offset is used for calculation.

In conjunction with the first aspect, in a second possible implementation of the first aspect, the OIS motor sensitivity calibration parameter is determined according to the steps of: pushing the OIS motor through the OIS controller, moving the lens to a designated position; waiting for the OIS motor Photographing after stabilization; when the captured image reaches a preset number of sheets, detecting feature point coordinates of each image, and according to the specified position of the lens and the respective images The feature point coordinates of the image determine the OIS motor sensitivity calibration parameters. The OIS motor sensitivity calibration parameter is stored therein before the terminal device leaves the factory, so that when the terminal device is shipped, the scene depth of the target scene is calculated, and the stored OIS motor sensitivity calibration parameter is used to convert the lens offset into an image. Offset.

With reference to the first aspect, in a third possible implementation manner of the first aspect, the lens offset is converted into an image offset according to the following formula,

Δx≈α×ΔC

Where Δx is the image offset, α is the OIS motor sensitivity calibration parameter, ΔC is the lens offset, and the lens offset is converted to the image offset by the OIS motor sensitivity calibration parameter, and the unit of the camera calibration parameter is maintained. Consistent.

In conjunction with the first aspect, in a fourth possible implementation manner of the first aspect, when the first camera is provided with an OIS system, the first camera calibration parameter after compensation and/or the second camera calibration after compensation And the first image and the second image obtained by the first camera and the second camera at the same time, and the scene depth of the target scene is specifically calculated by:

Use the following formula to determine the depth of the scene of the target scene.

Where Z is the scene depth of the target scene, f is the focal length, a ₁ is the OIS motor sensitivity calibration parameter of the first camera, Δ ₁ is the lens offset, and pixel pitch is the size of a pixel, a ₁ ×Δ ₁ ≈Δx ₁ is the first image offset, a × Δ ₁ × pixel pitch is the unit of the first image offset is converted from pixel to mm, and the main point of the first camera is changed from u ₁ ' to u ₁ after compensation. from B 'becomes B _1, the principal point of the second camera is u _2, x ₁ is the first image forming point, x ₂ is an imaging point of the second image.

In conjunction with the first aspect, in a fifth possible implementation manner of the first aspect, when the first camera and the second camera both have an OIS system, the first camera is calibrated according to the compensated parameter and/or compensated Second camera calibration parameter, and the first camera and the second camera are The first image and the second image obtained by the target scene are collected at the same time, and the calculation of the scene depth of the target scene includes:

a ₁ is the OIS motor sensitivity calibration parameter of the first camera, and Δ ₁ is the mirror of the first camera

among them,

The head offset, a ₁ × Δ ₁ ≈ Δx ₁ is the first image shift amount, and a ₁ × Δ ₁ × pixel pitch is a unit for converting the first image shift amount from pixel to mm, and a ₂ is second. The OIS motor sensitivity calibration parameter of the camera, Δ ₂ is the lens offset of the second camera, a ₂ × Δ ₂ ≈ Δx ₂ is the second image offset, and a ₂ × Δ ₂ × pixel pitch is the second image bias The unit pixel of the shift amount is converted to mm, and the main point of the first camera is changed from u ₁ ' to u ₁ after compensation, and the main point of the second camera is changed from u ₂ ' to u ₂ after compensation, and x ₁ is the first The image is imaged and x ₂ is the imaged point of the second image.

In a second aspect, an embodiment of the present invention provides a scenario depth calculation device, where the device includes: a first acquisition unit, a processing unit, and a calculation unit; and the first acquisition unit is configured to acquire a camera with an OIS system. a lens shift amount; wherein the first camera and/or the second camera are provided with an OIS system, the first camera and the second camera are juxtaposed on the body of the same terminal device; and the processing unit is configured to be preset according to The OIS motor sensitivity calibration parameter converts the lens offset into an image offset; the calculating unit is configured to perform calibration parameters according to the compensated first camera and/or compensated second camera calibration parameters, and The first camera and the second camera collect the first image and the second image respectively obtained by the target scene at the same time, and calculate the scene depth of the target scene; wherein, the first offset is compensated according to the lens offset of the first camera The calibration parameter of the camera compensates the calibration parameter of the second camera according to the lens offset of the second camera. By compensating for the change of the camera calibration parameters caused by the jitter of the terminal device, the parallax problem is solved, and the scene depth is calculated by using the compensated camera calibration parameters, and the calculated scene depth value is more accurate.

With reference to the second aspect, in a first possible implementation manner of the second aspect, the device further includes: a second acquiring unit, where the second acquiring unit is configured to: acquire the jitter of the terminal device detected by the gyro sensor Angular velocity information; converting the angular velocity information into a jitter amplitude of the terminal device, Driving the OIS motor to drive lens movement of the lens of the first camera and/or the second camera according to the amplitude of the shake, and acquiring a lens shift amount of the first camera and/or the second camera. Wherein, because the terminal device jitter time and the exposure time are inconsistent, and the jitter time is greater than the exposure time, multiple lens offsets are acquired, and one lens offset is determined from the plurality of lens offsets according to a preset rule. And when the scene depth of the target scene is subsequently calculated, the determined lens offset is used for calculation.

With reference to the second aspect, in a second possible implementation manner of the second aspect, the device further includes: a determining unit, where the determining unit is specifically configured to: push the OIS motor through the OIS controller, and move the lens to the designated Positioning; waiting for the OIS motor to stabilize after taking a picture; when the captured image reaches a preset number of sheets, detecting feature point coordinates of each image, and determining an OIS motor sensitivity calibration parameter according to the specified position of the lens. The OIS motor sensitivity calibration parameter is stored therein before the terminal device leaves the factory, so that when the terminal device is shipped, the scene depth of the target scene is calculated, and the stored OIS motor sensitivity calibration parameter is used to convert the lens offset into an image. Offset.

With reference to the second aspect, in a third possible implementation manner of the second aspect, the processing unit is specifically configured to:

Convert the lens offset to the image offset according to the following formula,

Δx≈α×ΔC

With reference to the second aspect, in a fourth possible implementation manner of the second aspect, when the first camera is provided with an OIS, the calculating unit is specifically configured to:

With reference to the second aspect, in a fifth possible implementation manner of the second aspect, when the first camera and the second camera both carry the OIS, the calculating unit is specifically configured to:

among them,

In a third aspect, an embodiment of the present invention provides a terminal, where the terminal includes a first camera and a second camera, where the first camera and the second camera are used to collect at least one target scene at a same time, respectively obtaining a first image. And a second image; wherein the first camera and/or the second camera are provided with an OIS system, the first camera and the second camera are juxtaposed on the body of the same terminal device; and the memory is configured to store the first An image and a second image; the processor is configured to acquire a lens offset of the camera with the OIS motor, and convert the lens offset into an image offset according to a preset OIS motor sensitivity calibration parameter Calculating a scene depth of the target scene according to the compensated first camera calibration parameter and/or the compensated second camera calibration parameter, and the first image and the second image acquired from the memory; wherein, according to First camera The lens offset compensates the calibration parameter of the first camera, and the calibration parameter of the second camera is compensated according to the lens offset of the second camera. The lens offset is used to compensate for the change of the camera calibration parameters caused by the jitter of the terminal device, and the parallax problem is solved. The compensated camera calibration parameters are used to calculate the scene depth of the target scene, and the calculated scene depth value is more accurate.

With reference to the third aspect, in a first possible implementation manner of the third aspect, the OIS system is specifically configured to: acquire angular velocity information of terminal device jitter detected by a gyro sensor; and convert the angular velocity information into a terminal device The amplitude of the jitter, driving the OIS motor to drive the lens movement of the lens of the first camera and/or the second camera according to the amplitude of the shake, and acquiring the lens offset of the first camera and/or the second camera. Wherein, because the terminal device jitter time and the exposure time are inconsistent, and the jitter time is greater than the exposure time, multiple lens offsets are acquired, and one lens offset is determined from the plurality of lens offsets according to a preset rule. And when the scene depth of the target scene is subsequently calculated, the determined lens offset is used for calculation.

In conjunction with the third aspect, in a second possible implementation manner of the third aspect, the processor is further configured to: by the OIS controller, push the OIS motor to move the lens to the designated position; wait for the OIS motor to stabilize Taking a picture; detecting a feature point coordinate of each image when the captured image reaches a preset number of sheets, and determining an OIS motor sensitivity calibration parameter according to the specified position of the lens and the feature point coordinates of each image; The memory is further configured to store the OIS motor sensitivity calibration parameters. The OIS motor sensitivity calibration parameter is stored therein before the terminal leaves the factory, so that when the terminal is calculated and the scene depth of the target scene is calculated, the stored OIS motor sensitivity calibration parameter is used to convert the lens offset into an image offset. the amount.

With reference to the third aspect, in a third possible implementation manner of the third aspect, the processor is specifically configured to convert the lens offset into an image offset according to the following formula,

Δx≈α×ΔC

With reference to the third aspect, in a fourth possible implementation manner of the third aspect, when the first camera is provided with an OIS system, the processor is specifically configured to determine a scene depth of the target scene by using:

Where Z is the scene depth of the target scene, f is the focal length, a ₁ is the OIS motor sensitivity calibration parameter of the first camera, Δ ₁ is the lens offset, pixel pitch is the size of a pixel, a ₁ ×Δ ₁ ≈Δx ₁ is the first image shift amount, and a×Δ ₁ ×pixel pitch is a unit for converting the first image shift amount from pixel to mm, and the main point of the first camera is changed from u ₁ ' to u ₁ after compensation. , the baseline B 'becomes B _1, the principal point of the second camera is u _2, x ₁ is the first image forming point, x ₂ of the second imaging point image.

With reference to the third aspect, in a fifth possible implementation manner of the third aspect, when the first camera and the second camera both have an OIS system, the processor is specifically configured to determine the target scenario by using the following formula: Scene depth,

among them,

DRAWINGS

The technical solutions of the embodiments of the present invention are further described in detail below with reference to the accompanying drawings and embodiments. Said.

Figure 1 is a block diagram showing the working principle of the OIS system;

Figure 2 is a block diagram of a depth calculation system;

FIG. 3 is a flowchart of a method for calculating a depth of a scene according to Embodiment 1 of the present invention;

Figure 4a is a schematic diagram of a lens offset scene;

Figure 4b is a schematic diagram of imaging changes before and after lens shift;

Figure 4c is a flow chart for determining the OIS motor sensitivity calibration parameters;

FIG. 5a is a schematic diagram of scene depth calculation according to an embodiment of the present invention; FIG.

FIG. 5b is still another schematic diagram of scene depth calculation according to an embodiment of the present invention;

Figure 6a is a schematic diagram of an image taken when compensating the calibration parameters of the dual camera;

Figure 6b is a schematic diagram of an image taken after the calibration parameters of the dual camera are not compensated;

Figure 6c is a partial enlarged view of Figure 6a;

Figure 6d is a partial enlarged view of Figure 6b;

Figure 7a is a schematic diagram of the depth of the scene when the calibration parameters of the dual camera are not compensated;

Figure 7b is a schematic diagram of the depth of the scene after compensating the calibration parameters of the dual camera;

FIG. 8 is a schematic structural diagram of a scene depth calculation apparatus according to Embodiment 2 of the present invention;

FIG. 9 is a schematic structural diagram of still another scene depth computing apparatus according to Embodiment 2 of the present invention;

FIG. 10 is a schematic structural diagram of a terminal according to Embodiment 3 of the present invention.

detailed description

In the present invention, the terminal device may be a device having a dual camera, including but not limited to a camera (such as a digital camera), a video camera, a mobile phone (such as a smart phone), a tablet (Pad), and a personal digital assistant (Personal). The digital assistant (PDA), the portable device (for example, a portable computer), the wearable device, and the like are not specifically limited in the embodiment of the present invention.

Referring to FIG. 1 , the terminal device may be a mobile phone, and the following uses a mobile phone as an example to perform an embodiment of the present invention. set forth.

At present, more and more mobile phones use a dual camera to calculate the depth of the scene. The dual camera simulates the human binocular vision principle to perceive the distance, that is, observing an object from two points, and acquiring images at different viewing angles, according to the pixels between the images. The matching relationship is obtained by calculating the offset between pixels by the principle of triangulation to obtain the depth of the scene of the object. When one or two of the dual cameras are equipped with the OIS system, the OIS causes the lens to shift, which causes the dual camera calibration parameters to change, resulting in parallax problems, which in turn results in inaccurate scene depth calculation. Therefore, it is necessary to compensate the dual camera calibration parameters. , so that the scene depth of the target scene is calculated accurately.

Figure 1 is a block diagram of the working principle of the OIS system. As shown in FIG. 1, the terminal device includes an OIS system 100 and an Image Signal Processor (ISP) 110. The OIS system 100 includes an OIS controller 120, a gyro sensor 130, a Hall sensor 140, a motor 150, and a camera 160.

The camera 160 includes a first camera and a second camera. The first camera and the second camera may be juxtaposed in front of the terminal device, or may be juxtaposed on the back of the terminal device, and may be arranged in a horizontal arrangement or a vertical arrangement. The first camera and/or the second camera are provided with an OIS system, and the first camera and the second camera respectively have lenses (not shown in FIG. 1).

The Hall sensor 140 is a magnetic field sensor that performs displacement measurement based on the Hall effect for acquiring the lens shift amount of the camera with the OIS system, that is, the lens shift amount of the first camera and/or the second camera.

The gyro sensor 130 is a positioning system based on the movement of the terminal device in a free space orientation for acquiring angular velocity information when the terminal device is shaken.

The OIS controller 120 acquires angular velocity information from the gyro sensor 130, converts the angular velocity information into a jitter amplitude of the terminal device, and transmits the jitter amplitude as a reference signal to the motor 150.

The motor 150 may be an OIS motor for driving the lens movement of the camera with the OIS system according to the amplitude of the shake to ensure the sharpness of the image; wherein the movement refers to moving in the X and/or Y direction, and the Y direction refers to the lens. One direction in the plane of the line connecting the optical center and the focus, the X direction refers to the direction of the light passing through the lens and perpendicular to the Y direction.

The OIS controller 120 also acquires the first image and the second image obtained by acquiring the target scene at the same time from the first camera and the second camera.

The ISP 110 stores the lens shift amount, the first image, and the second image acquired from the OIS controller 120.

The working principle of the OIS system will be specifically described below in conjunction with the various components in FIG.

At the time of preparation for shooting, the terminal device performs initialization, and usually, when ready, the OIS controller 120 controls the shutter to acquire an image.

At the time of shooting, the terminal device may shake, and the OIS controller 120 reads the angular velocity information detected by the gyro sensor 130, converts the angular velocity information into the jitter amplitude of the terminal device, and transmits it as a reference signal to the OIS motor, and the OIS motor according to the jitter amplitude Move the lens of the camera with OIS system to avoid blurring of the captured image caused by the jitter of the terminal device and ensure the sharpness of the image. Wherein, the movement may be that the lens of the first camera moves in the X and/or Y direction and/or the lens of the second camera moves in the X and/or Y direction. The OIS controller 120 reads the lens offset of the camera with the OIS system detected by the Hall sensor 140, that is, the lens offset of the first camera and/or the second camera, and acquires the captured image from the camera. That is, the first image and the second image obtained by the target scene are acquired at the same time corresponding to the first camera and the second camera, respectively, and the lens offset and the captured image are sent to the ISP 110. The ISP 110 stores the lens shift amount and the first image and the second image captured by the camera.

Since the terminal device jitter time is generally greater than its exposure time, for example, the terminal device jitter duration is 30ms and the exposure time is 2ms. When the terminal device shakes, the Hall sensor 140 acquires 15 lens offsets, OIS. The controller 120 reads the 15 lens offsets from the Hall sensor 140 according to a preset rule, and determines a lens offset from the 15 lens offsets, and uses in the subsequent process. The determined lens offset is used as the lens offset described in the context to perform scene depth calculation of the target scene.

Figure 2 is a block diagram of the depth calculation system. As shown in FIG. 2, the depth calculation system includes an ISP 110 and a depth calculation module 210. The depth calculation module 210 acquires preset calibration information from the ISP 110 and The ISP 110 reads the stored OIS information, and the first camera and the second camera acquire the first image and the second image respectively obtained by the target scene at the same time, calculate the scene depth of the target scene, and output the disparity map/depth map. Wherein, the calibration information is the camera calibration parameters such as focal length, baseline, optical center and principal point when initializing; the OIS information is the lens offset.

Specifically, the depth calculation module acquires the lens offset. Since the unit of the lens offset is code and the unit of the scene depth is millimeter (mm), the two are inconsistent. Therefore, the lens needs to be offset according to the OIS motor sensitivity. The quantity is converted into image offset, the unit is pixel (pixel), and then the camera offset parameter is compensated by the lens offset, and the scene depth value of the target scene is calculated according to the compensated camera calibration parameter, thereby calculating the scene depth value. More precise.

FIG. 3 is a flowchart of a method for calculating a scene depth according to Embodiment 1 of the present invention. As shown in FIG. 3, the method includes:

S310, obtaining the lens shift amount (see FIG. 4b [Delta] C, Δ Figure 5a _1, Δ _1, and FIG. 5b Δ _2).

Specifically, the lens shift amount can be obtained by Hall sensor detection.

S320, whether the lens offset is abnormal; if it is not abnormal, jump to S340; if abnormal, jump to S380.

Specifically, if the lens offset is greater than the preset threshold, the lens offset is abnormal; if the lens offset is not greater than the preset threshold, the lens offset is not abnormal.

S340, converting the lens offset into an image offset (see Δx of FIG. 4b). Before executing S340, S330 needs to be executed in advance, that is, the OIS motor sensitivity calibration parameter, that is, the image offset caused by the unit lens offset, is input, wherein each camera with the OIS system has its corresponding OIS motor sensitivity calibration. Parameters, the OIS motor sensitivity calibration parameters have been pre-stored before the terminal device leaves the factory. Using the OIS motor sensitivity calibration parameters, the lens offset can be converted to an image offset.

S350, compensates for dual camera calibration parameters.

Specifically, due to the jitter of the terminal device during photographing, some calibration parameters (such as the optical center, the main point, the baseline, etc.) of the camera are changed, the image offset is calculated according to the lens offset, and the lens offset is used to compensate for the change. The camera calibration parameters determine the value of the camera calibration parameters after the change. Referring to Fig. 4, the optical center changes from C' to C, and the principal point changes from u' to u. Referring to Fig. 5a, the optical center of the first camera lens changes from C ₁ ' to C ₁ , the main point changes from u ₁ ' to u ₁ , and the baseline changes from B' to B ₁ . Referring to Fig. 5b, the optical center of the first camera lens changes from C ₁ ' to C ₁ , the main point changes from u ₁ ' to u ₁ , and the optical center of the second camera lens changes from C ₂ ' to C ₂ , the main point From u ₂ ' to u ₂ , the baseline changes from B' to B ₂ .

S380: Calculate a scene depth of the target scene. See Equation 2 and Equation 4 for the calculation formula. It is necessary to perform S360 in advance before executing S380, that is, input the first image, and S370, that is, input the second image.

Specifically, the scene depth of the target scene is determined according to the compensated camera calibration parameters, the first image, and the second image. The first camera and the second camera acquire the first image and the second image respectively obtained by the target scene at the same time.

S390: Output a scene depth of the target scene.

How to determine the OIS motor sensitivity calibration parameters is described below in conjunction with Figures 4a-4c.

Figure 4a is a schematic diagram of a lens offset scene. As shown in Fig. 4a, taking the OIS motor to push the lens movement of a camera as an example, the OIS motor pushes the lens from the position of the elliptical dotted line to the specified position (x _i , y _i ), respectively, to take an image of the fixed chart, and the image sensor will The optical image acquired by the camera is converted into an electronic signal, and the lens offset can be determined according to the position before and after the lens is moved, and the image offset is determined according to the images of the two fixed charts.

Figure 4b is a schematic diagram of imaging changes before and after lens shift. As shown in Fig. 4b, the OIS motor pushes the lens of a camera in the X direction as an example. Before the lens moves, the camera calibration parameters: the focal length is f, the optical center is C', and the main point is u'. After the lens is moved, part of the calibration parameters of the camera changes: the optical center changes from C' to C, and the main point changes from u' to u. The imaging points before and after the lens movement are x' and x, respectively, ΔC is the distance between the optical center C' and the optical center C, that is, the lens shift amount, the unit is code, Δx is the distance between the imaging point x' and the imaging point x, that is, the image Offset in pixels. According to the lens offset and the image offset, the image offset caused by the unit lens offset can be measured, that is, the OIS motor sensitivity calibration parameter. According to the OIS motor sensitivity calibration parameter, the actual image offset can be calculated based on the lens offset during subsequent shooting to compensate for the camera calibration parameters when the terminal is shaken.

When determining the OIS motor sensitivity calibration parameters, it is assumed that between the lens shift amount ΔC and the image shift amount Δx The relationship is linear, and the OIS motor sensitivity calibration parameters can be obtained as: α ≈ Δx / ΔC, and the α unit is pixels/code.

However, ΔC and Δx are not strictly linear. Higher-order models, such as second-order, third-order or higher, can be used to capture more images for more accurate OIS motor sensitivity calibration parameters.

Figure 4c is a flow chart for determining OIS motor sensitivity calibration parameters. As shown in Figure 4c, it includes:

S410, pushing the OIS motor through the OIS controller to move the lens to the designated position (x _i , y _i ).

Specifically, it is possible to move the coordinates of the lens from the (x ₁ , y ₁ ) position to the (x _i , y _i ) position.

S420, waiting for the OIS motor to stabilize and take a picture.

S430, whether the number of captured images reaches a preset number of sheets; if not, jump to S410; enough, jump to S440.

Specifically, if the captured image is too small, the determined OIS motor sensitivity calibration parameter may have a relatively large error, and multiple images may be taken to improve the accuracy of the OIS motor sensitivity calibration parameter.

In addition, there is not necessarily a strict linear relationship between the lens shift amount and the image shift amount. In order to more accurately determine the image shift amount under a certain lens shift amount, it is necessary to image the image at different lens shift amounts. The offset is measured.

S440, detecting feature point coordinates in each image, and determining the OIS motor sensitivity calibration parameter in combination with the lens position (x _i , y _i ).

Specifically, the feature point coordinates in the captured image before and after the lens movement are respectively detected, and the image offset amount is acquired. The lens shift amount is determined according to the moving distance of the lens, and the OIS motor sensitivity calibration parameter is determined according to the image shift amount and the lens shift amount.

S450: Store the OIS motor sensitivity calibration parameter into the terminal device.

Specifically, before the terminal device leaves the factory, the OIS motor sensitivity calibration parameter is stored therein, so that after the terminal device is shipped from the factory, when the target device is used to collect the target scene, the lens is adjusted according to the OIS motor sensitivity calibration parameter stored in advance. The offset is converted to the image offset, and the camera offset parameter is compensated by the lens offset to make the scene depth of the target scene more accurate.

The depth calculation will be further described below with reference to Figs. 5a, 5b, 6a-6c.

In an example, as shown in FIG. 5a, FIG. 5a is a schematic diagram of scene depth calculation according to an embodiment of the present invention. The first camera has an OIS system, the second camera does not have an OIS system, and the lens of the first camera moves in the X direction. The lens of the camera is a convex lens, and the incoming rays and the corresponding and parallel rays are formed. The conjugate ray, the intersection of the line connecting the incident point and the exit point with the main optical axis, is called the focal point of the convex lens. The distance from the focus plane or the imaging plane such as CCD is called the focal length. The point at the center of the lens is called the optical center. The intersection of the line of sight with the imaging plane such as the film or CCD is called the main point, and the distance between the first camera lens and the second camera lens is called the baseline. At initialization, the following calibration parameters of the camera are known: the focal length is f, the optical center of the lens of the first camera is C ₁ ', the main point is u ₁ ', and the optical center of the lens of the second camera is C ₂ , the main The point is u ₂ and the baseline is B'. Wherein, when the shutter is closed, the imaging point of the first image acquired by the first camera is x ₁ , and the imaging point of the second image acquired by the second camera is x ₂ .

At this point, using the similar triangle principle, the calculated scene depth Z' is:

In the prior art, since the partial calibration parameters of the camera caused by the jitter of the terminal device are changed when the underarm shutter is not considered, the scene depth calculation is based on the uncompensated camera calibration parameters, the main point u ₁ ' and the baseline B', To complete. The calculated scene depth Z' error is large.

When pressed the shutter, the terminal apparatus can cause jitter calibration parameters of the camera portion is changed, pushing the first camera lens according to the jitter amplitude shift occurs, the OIS motor, a first optical center of the camera lens from C ₁ 'becomes C ₁ ( That is, the lens offset of the first camera is in code), the main point changes from u ₁ ' to u ₁ , and the baseline changes from B' to B ₁ , and u ₁ and B ₁ need to be calculated. The camera calibration parameters of the lens offset compensation are used to determine the value of the compensated camera calibration parameters. Using the similar triangle principle, the calculated scene depth Z is:

Where a ₁ is the OIS motor sensitivity calibration parameter of the first camera, Δ ₁ is the lens offset of the first camera, pixel pitch is the size of one pixel, and a ₁ × Δ ₁ ≈ Δx ₁ is the first image offset In units of pixels, a ₁ × Δ ₁ × pixel pitch is a unit for converting the first image shift amount from pixel to mm.

In another example, as shown in FIG. 5b, FIG. 5b is still another schematic diagram of scene depth calculation according to an embodiment of the present invention. The first camera and the second camera are simultaneously provided with an OIS system, and the first camera and the second camera are simultaneously moved in the X direction. When initializing, the following calibration parameters of the camera are known: the focal length is f, the first camera The optical center of the lens is C ₁ ', the main point is u ₁ ', the optical center of the lens of the second camera is C ₂ ', the main point is u ₂ ', and the baseline is B'. When the shutter is depressed, the imaging point of the first image acquired by the first camera is x ₁ , and the imaging point of the second image acquired by the second camera is x ₂ .

In the prior art, since the partial calibration parameters of the camera caused by the jitter of the terminal device are changed when the underarm shutter is not considered, the depth calculation is based on the uncompensated camera calibration parameters, the main points u ₁ ', u ₂ ' and the baseline. B' to complete. The calculated scene depth Z' error is large.

When the shutter is lowered, the terminal device shake will cause some calibration parameters of the camera to change, that is, the optical center of the first camera lens changes from C ₁ ' to C ₁ , and the main point changes from u ₁ ' to u ₁ , the second camera The optical center of the lens changes from C ₂ ' to C ₂ (corresponding to the lens offset of the second camera, the unit is code), the main point changes from u ₂ ' to u ₂ , and the baseline changes from B' to B ₂ , which needs to be calculated. u ₁ , u ₂ and B _{2 are} produced. Using the lens offset (including the lens offset of the first camera and the lens offset of the second camera), compensating for the changed camera calibration parameters, determining the value of the camera calibration parameter after the change, using the similar triangle principle, calculating The depth Z of the scene is:

Where a ₁ is the OIS motor sensitivity calibration parameter of the first camera, Δ ₁ is the lens offset of the first camera, and a ₁ × Δ ₁ ≈ Δx ₁ is the first image offset, a ₁ × Δ ₁ × pixel The pitch is to convert the unit of the first image offset from pixel to mm, a ₂ is the OIS motor sensitivity calibration parameter of the second camera, and Δ ₂ is the lens offset of the second camera, a ₂ × Δ ₂ ≈ Δx ₂ For the second image shift amount, a ₂ × Δ ₂ × pixel pitch is a unit pixel of the second image shift amount converted into mm.

Although FIG. 5a and FIG. 5b illustrate how the camera calibration parameter compensation is performed by taking the movement of the camera in one direction as an example, it should be realized that the same can realize the compensation of the camera calibration parameters of the two cameras moving in two directions. I won't go into details here.

Figure 6a is a schematic diagram of an image taken when compensating the calibration parameters of the dual camera; Figure 6b is a schematic diagram of the image taken after the calibration parameters of the dual camera are not compensated; Figure 6c is a partial enlarged view of Figure 6a; Figure 6d is a partial enlarged view of Figure 6b It can be seen from Fig. 6a-6d that the image alignment effect is poor when the dual camera calibration parameters are not compensated, and the alignment effect of the image is good after compensating the dual camera calibration parameters.

Figure 7a is a schematic diagram of the depth of the scene when the calibration parameters of the dual camera are not compensated; Figure 7b is a schematic diagram of the depth of the scene after the calibration parameters of the dual camera are compensated. In Figures 7a and 7b, different depth values are represented by different colors, and black indicates that the scene depth cannot be calculated. In Figure 7a, the depth of the scene measured at 1000 mm is 1915.8 mm and the depth of the scene measured at 300 mm is 344.6 mm. In Figure 7b, the depth of the scene measured at 1000 mm is 909.6 mm and the depth of the scene measured at 300 mm is 287.4 mm. It follows that the calculated scene depth value is more accurate after compensating the dual camera calibration parameters.

The scene depth calculation method with the dual camera terminal provided by the embodiment of the invention solves the problem that the scene depth value is inaccurate due to the parallax problem.

FIG. 8 is a schematic structural diagram of a scene depth calculation apparatus according to Embodiment 2 of the present invention. As shown in FIG. 8, the scene depth calculation apparatus 800 includes a first acquisition unit 810, a processing unit 820, and a calculation unit 830.

The first acquiring unit 810 is configured to acquire a lens offset of the camera with the OIS system; wherein the first camera and/or the second camera have an OIS system, and the first camera and the second camera are arranged side by side in the same On the body of the terminal device.

The processing unit 820 is configured to convert the lens offset into an image offset according to a preset OIS motor sensitivity calibration parameter.

The calculating unit 830 is configured to obtain, according to the compensated first camera calibration parameter and/or the compensated second camera calibration parameter, and the acquired target scenes obtained by the first camera and the second camera at the same time The first image and the second image are used to calculate a scene depth; wherein the calibration parameter of the first camera is compensated according to the lens offset of the first camera, and the calibration parameter of the second camera is compensated according to the lens offset of the second camera.

Specifically, the processing unit 820 is specifically configured to: convert the lens offset into an image offset according to the following formula,

Δx≈α×ΔC

Where Δx is the image shift amount, α is the OIS motor sensitivity calibration parameter, and ΔC is the lens shift amount.

When the first camera has an OIS system and the second camera does not have an OIS system, the calculating unit 830 is specifically configured to:

When both the first camera and the second camera have an OIS system, the calculating unit 830 is specifically configured to:

among them,

FIG. 9 is a schematic structural diagram of still another scene depth computing apparatus according to Embodiment 2 of the present invention. As shown in FIG. 9 , the device may also be a scene depth computing device 900. The device 900 may further include: a second acquiring unit 910 and a determining unit 920.

a second obtaining unit 910, configured to acquire angular velocity information of the terminal device jitter detected by the gyro sensor; convert the angular velocity information into a jitter amplitude of the terminal device, and drive the OIS motor to push the lens of the first camera according to the jitter amplitude And/or the lens of the second camera moves and acquires the lens offset of the first camera and/or the second camera.

The determining unit 920 is configured to: push the OIS motor through the OIS controller, move the lens to the designated position; wait for the OIS motor to stabilize and take a picture; and when the captured image reaches a preset number of sheets, detect the feature point coordinates of each image And determining the OIS motor sensitivity calibration parameter according to the specified position of the lens.

FIG. 10 is a schematic structural diagram of a terminal according to Embodiment 3 of the present invention. As shown in FIG. 10, the terminal 1000 includes a camera 1010 (the camera 1010 includes a first camera and a second camera) processor 1020, a memory 1030, and a system bus; the camera 1010, the processor 1020, and the memory 1030 establish a connection through a system bus.

The first camera and the second camera are configured to acquire at least the target scene at the same time to obtain the first image and the second image respectively; wherein the first camera and/or the second camera have an OIS system, the first camera and the second camera Parallel to the fuselage of the same terminal device.

The memory 1020 is configured to store the first image and the second image.

The processor 1030 is configured to acquire a lens offset of the camera with the OIS motor, and convert the lens offset into an image offset according to a preset OIS motor sensitivity calibration parameter; a first camera calibration parameter and/or a compensated second camera calibration parameter, and a first image and a second image acquired from the memory, calculating a scene depth of the target scene; wherein, according to the lens offset of the first camera The calibration parameter of the first camera is compensated, and the calibration parameter of the second camera is compensated according to the lens offset of the second camera.

Specifically, the OIS system is specifically configured to acquire angular velocity information of the terminal device jitter detected by the gyro sensor, convert the angular velocity information into a jitter amplitude of the terminal device, and drive the OIS motor to push the first camera according to the jitter amplitude. The lens of the lens and/or the second camera moves and acquires the lens offset of the first camera and/or the second camera.

The processor 1030 is further configured to: push the OIS motor through the OIS controller, move the lens to a designated position; wait for the OIS motor to stabilize and take a picture; and when the captured image reaches a preset number of sheets, detect feature points of each image Coordinates, and determining OIS motor sensitivity calibration parameters according to the specified position of the lens and the feature point coordinates of the respective images.

The memory 1020 is further configured to store the OIS motor sensitivity calibration parameter.

Further, the processor 1030 is specifically configured to convert the lens offset into an image offset according to the following formula.

Δx≈α×ΔC

Further, when the first camera is provided with an OIS system, the processor is specifically configured to determine a scene depth of the target scene by using the following formula:

Further, when both the first camera and the second camera are provided with an OIS system, the processor is specifically configured to determine a scene depth of the target scene by using the following formula:

among them,

A person skilled in the art should further appreciate that the elements and algorithm steps of the various examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both, in order to clearly illustrate hardware and software. Interchangeability, the composition and steps of the various examples have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the solution. A person skilled in the art can use different methods for implementing the described functions for each particular application, but such implementation should not be considered to be beyond the scope of the present invention.

It will be understood by those skilled in the art that all or part of the steps of implementing the above embodiments may be performed by a program, and the program may be stored in a computer readable storage medium, which is non-transitory ( English: non-transitory) media, such as random access memory, read-only memory, flash memory, hard disk, solid state disk, magnetic tape (English: magnetic tape), floppy disk (English: floppy disk), CD (English: optical disc) And any combination thereof.

Claims

A method for calculating a depth of a scene, the method comprising:

Obtaining a lens offset of the camera with the OIS system; wherein the first camera and/or the second camera are provided with an OIS system, and the first camera and the second camera are juxtaposed on the body of the same terminal device;

Converting the lens offset into an image offset according to a preset OIS motor sensitivity calibration parameter;

And the first camera and the second image obtained by the first camera and the second camera respectively acquiring the target scene at the same time according to the compensated first camera calibration parameter and/or the compensated second camera calibration parameter, Calculating a scene depth of the target scene; wherein, the calibration parameter of the first camera is compensated according to the lens offset of the first camera, and the calibration parameter of the second camera is compensated according to the lens offset of the second camera.
The method of claim 1 further comprising:

Obtaining angular velocity information of terminal device jitter detected by the gyro sensor;

Converting the angular velocity information into a jitter amplitude of the terminal device, driving the OIS motor to drive the lens movement of the lens of the first camera and/or the second camera according to the jitter amplitude, and acquiring the first camera and/or the second camera The lens offset.
The method of claim 1 wherein the OIS motor sensitivity calibration parameters are determined according to the following steps:

Push the OIS motor through the OIS controller to move the lens to the specified position;

Waiting for the OIS motor to stabilize after taking a picture;

When the captured image reaches a preset number of sheets, the feature point coordinates of each image are detected, and the OIS motor sensitivity calibration parameter is determined according to the specified position of the lens and the feature point coordinates of the respective images.
The method according to claim 1, wherein the converting the lens offset into an image offset according to a preset OIS motor sensitivity calibration parameter comprises:

Convert the lens offset to the image offset according to the following formula,

Δx≈α×ΔC

Where Δx is the image shift amount, α is the OIS motor sensitivity calibration parameter, and ΔC is the lens shift amount.
The method according to claim 1, wherein when the first camera is provided with an OIS system, the compensated first camera calibration parameter and/or the compensated second camera calibration parameter, and the A camera and the second camera collect the first image and the second image respectively obtained by the target scene at the same time, and calculating the scene depth of the target scene specifically includes:

Determining the depth of the scene of the target scene by using the following formula,

Where Z is the scene depth of the target scene, f is the focal length, a 1 is the OIS motor sensitivity calibration parameter of the first camera, Δ 1 is the lens offset, pixel pitch is the size of a pixel, a 1 ×Δ 1 ≈Δx 1 is the first image shift amount, and a×Δ 1 ×pixel pitch is a unit for converting the first image shift amount from pixel to mm, and the main point of the first camera is changed from u 1 ' to u 1 after compensation. , the baseline B 'becomes B 1, the principal point of the second camera is u 2, x 1 is the first image forming point, x 2 of the second imaging point image.
The method according to claim 1, wherein when the first camera and the second camera both have an OIS system, the compensated first camera calibration parameter and/or the compensated second camera calibration parameter And the first image and the second image obtained by the first camera and the second camera at the same time, and the scene depth of the target scene is specifically calculated by:

Determining the depth of the scene of the target scene by using the following formula,

Where a 1 is the OIS motor sensitivity calibration parameter of the first camera, Δ 1 is the lens offset of the first camera, and a 1 × Δ 1 ≈ Δx 1 is the first image offset, a 1 × Δ 1 × pixel The pitch is to convert the unit of the first image offset from pixel to mm, a 2 is the OIS motor sensitivity calibration parameter of the second camera, and Δ 2 is the lens offset of the second camera, a 2 × Δ 2 ≈ Δx 2 For the second image offset, a 2 × Δ 2 × pixel pitch is to convert the unit pixel of the second image offset into mm, and the main point of the first camera is changed from u 1 ' to u 1 after compensation, after compensation The principal point of the second camera changes from u 2 ' to u 2 , x 1 is the first image imaging point, and x 2 is the imaging point of the second image.
A scene depth computing device, comprising: a first acquiring unit, a processing unit, and a calculating unit;

The first acquiring unit is configured to acquire a lens offset of the camera with the OIS system; wherein the first camera and/or the second camera have an OIS system, and the first camera and the second camera are juxtaposed in the same terminal On the fuselage of the device;

The processing unit is configured to convert the lens offset into an image offset according to a preset OIS motor sensitivity calibration parameter;

The calculating unit is configured to: according to the compensated first camera calibration parameter and/or the compensated second camera calibration parameter, and the first camera and the second camera respectively acquire the target scene at the same time An image and a second image are used to calculate a scene depth of the target scene; wherein, the calibration parameter of the first camera is compensated according to the lens offset of the first camera, and the calibration of the second camera is compensated according to the lens offset of the second camera parameter.
The method according to claim 7, wherein the device further comprises: a second acquiring unit;

The second obtaining unit is specifically configured to:

Obtaining angular velocity information of terminal device jitter detected by the gyro sensor;

Converting the angular velocity information into a jitter amplitude of the terminal device, driving the OIS motor to drive the lens movement of the lens of the first camera and/or the second camera according to the jitter amplitude, and acquiring the first camera and/or the second camera The lens offset.
The device according to claim 7, wherein the device further comprises: a determining unit;

The determining unit is specifically configured to: push the OIS motor through the OIS controller to move the lens to a designated position;

Waiting for the OIS motor to stabilize after taking a picture;

When the captured image reaches a preset number of sheets, the feature point coordinates of each image are detected, and the OIS motor sensitivity calibration parameter is determined according to the specified position of the lens.
The method according to claim 7, wherein the processing unit is specifically configured to:

Convert the lens offset to the image offset according to the following formula,

Δx≈α×ΔC

Where Δx is the image shift amount, α is the OIS motor sensitivity calibration parameter, and ΔC is the lens shift amount.
The method according to claim 5, wherein when the first camera is provided with an OIS system, the calculating unit is specifically configured to:

Determining the depth of the scene of the target scene by using the following formula,

Where Z is the scene depth of the target scene, f is the focal length, a 1 is the OIS motor sensitivity calibration parameter of the first camera, Δ 1 is the lens offset, pixel pitch is the size of a pixel, a 1 ×Δ 1 ≈Δx 1 is the first image shift amount, and a×Δ 1 ×pixel pitch is a unit for converting the first image shift amount from pixel to mm, and the main point of the first camera is changed from u 1 ' to u 1 after compensation. , the baseline B 'becomes B 1, the principal point of the second camera is u 2, x 1 is the first image forming point, x 2 of the second imaging point image.
The device according to claim 7, wherein when both the first camera and the second camera are provided with an OIS system, the calculating unit is specifically configured to:

Determining the depth of the scene of the target scene by using the following formula,

Where a 1 is the OIS motor sensitivity calibration parameter of the first camera, Δ 1 is the lens offset of the first camera, and a 1 × Δ 1 ≈ Δx 1 is the first image offset, a 1 × Δ 1 × pixel The pitch is to convert the unit of the first image offset from pixel to mm, a 2 is the OIS motor sensitivity calibration parameter of the second camera, and Δ 2 is the lens offset of the second camera, a 2 × Δ 2 ≈ Δx 2 For the second image offset, a 2 × Δ 2 × pixel pitch is to convert the unit pixel of the second image offset into mm, and the main point of the first camera is changed from u 1 ' to u 1 after compensation, after compensation The principal point of the second camera changes from u 2 ' to u 2 , x 1 is the first image imaging point, and x 2 is the imaging point of the second image.
A terminal, comprising: a first camera and a second camera, wherein the first camera and the second camera are configured to at least acquire a target scene at the same time, respectively obtaining a first image and a second image; wherein The first camera and/or the second camera are provided with an OIS system, and the first camera and the second camera are arranged side by side on the body of the same terminal device;

a memory for storing the first image and the second image;

a processor, configured to acquire a lens offset of a camera with an OIS motor, and convert the lens offset into an image offset according to a preset OIS motor sensitivity calibration parameter; a camera calibration parameter and/or a compensated second camera calibration parameter, and a first image and a second image acquired from the memory, calculating a scene depth of the target scene; wherein, according to a lens offset of the first camera The quantity compensates the calibration parameter of the first camera, and compensates the calibration parameter of the second camera according to the lens offset of the second camera.
The terminal according to claim 13, wherein the OIS system is specifically configured to acquire angular velocity information of terminal device jitter detected by the gyro sensor;

Converting the angular velocity information into a jitter amplitude of the terminal device, driving the OIS motor to drive the lens movement of the lens of the first camera and/or the second camera according to the jitter amplitude, and acquiring the first camera and/or the second camera The lens offset.
The terminal according to claim 13, wherein the processor is further configured to: push the OIS motor through the OIS controller to move the lens to a designated position;

Waiting for the OIS motor to stabilize after taking a picture;

When the captured image reaches a preset number of sheets, the feature point coordinates of each image are detected, and according to Determining an OIS motor sensitivity calibration parameter by the specified position of the lens and the feature point coordinates of the respective images;

The memory is further configured to store the OIS motor sensitivity calibration parameter.
The terminal according to claim 13, wherein the processor is specifically configured to:

Convert the lens offset to the image offset according to the following formula,

Δx≈α×ΔC

Where Δx is the image shift amount, α is the OIS motor sensitivity calibration parameter, and ΔC is the lens shift amount.
The terminal according to claim 13, wherein when the first camera is provided with an OIS system, the processor is specifically configured to determine a scene depth of the target scene by using the following formula:

Where Z is the scene depth of the target scene, f is the focal length, a 1 is the OIS motor sensitivity calibration parameter of the first camera, Δ 1 is the lens offset, pixel pitch is the size of a pixel, a 1 ×Δ 1 ≈Δx 1 is the first image shift amount, and a×Δ 1 ×pixel pitch is a unit for converting the first image shift amount from pixel to mm, and the main point of the first camera is changed from u 1 ' to u 1 after compensation. , the baseline B 'becomes B 1, the principal point of the second camera is u 2, x 1 is the first image forming point, x 2 of the second imaging point image.
The terminal according to claim 13, wherein when both the first camera and the second camera are provided with an OIS system, the processor is specifically configured to determine a scene depth of the target scene by using the following formula:

Where a 1 is the OIS motor sensitivity calibration parameter of the first camera, Δ 1 is the lens offset of the first camera, and a 1 × Δ 1 ≈ Δx 1 is the first image offset, a 1 × Δ 1 × pixel The pitch is to convert the unit of the first image offset from pixel to mm, a 2 is the OIS motor sensitivity calibration parameter of the second camera, and Δ 2 is the lens offset of the second camera, a 2 × Δ 2 ≈ Δx 2 For the second image offset, a 2 × Δ 2 × pixel pitch is to convert the unit pixel of the second image offset into mm, and the main point of the first camera is changed from u 1 ' to u 1 after compensation, after compensation. The principal point of the second camera changes from u 2 ' to u 2 , x 1 is the first image imaging point, and x 2 is the imaging point of the second image.