[go: up one dir, main page]
More Web Proxy on the site http://driver.im/

WO2021093804A1 - 全向立体视觉的摄像机配置系统及摄像机配置方法 - Google Patents

全向立体视觉的摄像机配置系统及摄像机配置方法 Download PDF

Info

Publication number
WO2021093804A1
WO2021093804A1 PCT/CN2020/128353 CN2020128353W WO2021093804A1 WO 2021093804 A1 WO2021093804 A1 WO 2021093804A1 CN 2020128353 W CN2020128353 W CN 2020128353W WO 2021093804 A1 WO2021093804 A1 WO 2021093804A1
Authority
WO
WIPO (PCT)
Prior art keywords
camera
camera lens
depth
lens
omnidirectional
Prior art date
Application number
PCT/CN2020/128353
Other languages
English (en)
French (fr)
Inventor
薛乐山
Original Assignee
南京深视光点科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 南京深视光点科技有限公司 filed Critical 南京深视光点科技有限公司
Publication of WO2021093804A1 publication Critical patent/WO2021093804A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/327—Calibration thereof

Definitions

  • the present invention relates to stereo vision (Stereo Vision) technology, in particular to a method that can arrange a plurality of camera lenses at intervals to form a rectangle, and arrange the lens centers of at least three camera lenses to be corrected on the same reference line, so as to obtain Unobstructed depth map, and then the "omnidirectional stereo vision camera configuration system and camera configuration method" can be obtained after image stitching is completed.
  • Stereo Vision Stereo Vision
  • the camera In binocular vision, the camera needs to be calibrated and corrected.
  • the purpose of camera rectification is to achieve an ideal binocular camera system so that the optical axes of at least two cameras are completely parallel (that is, the mirror centers are only between each other. There is an X component), and it is perpendicular to the baseline (the line between the center of the left and right cameras is the baseline), and then depth calculation and 3D reconstruction can be continued.
  • the lens centers (optical centers) of the left and right cameras before correction are not parallel, while the optical axes of the left and right cameras of the ideal binocular system after correction are parallel to each other, the optical axis and the image plane are perpendicular, and the imaging point is on the left and right images. Therefore, when performing stereo matching in subsequent stereo matching, you only need to search for the matching points of the left and right images in the search range of the same column, which can greatly improve the efficiency of camera correction.
  • Taiwanese invention patent No. TW201832547 “Image device for generating panoramic depth images, related methods and related image devices”
  • the Taiwan case mainly uses four ultra-wide-angle fisheye lenses (>190 degrees) to be arranged back and forth in pairs, and the extracted wide-angle images are projected to the equirectangular coordinate system, and the two left and right fish are located on the same side.
  • the eye lens can be used to calculate a 180x180 degree stereo depth image, and finally the depth images on both sides are stitched together to obtain a 360x180 degree omnidirectional depth image.
  • the baselines b12 and b13 are the connections between the first and second cameras before the correction in the U.S. case, and the lens centers O of the first and third cameras before the correction in the U.S. case, and the baselines b12' and b13' are the first and second cameras after the correction.
  • the baseline b13 is used as the reference line.
  • the first and third cameras only need to rotate to make the X axis Parallel to the vector of the baseline b13, although the second camera can also pass the offset matrix T( It can also be called a translation matrix) to make the X axis parallel to the vector of the baseline b12, but because the second camera deviates too far from the X axis, the y and z components of the offset matrix T are large, and when the y and z components reach a certain value (For example, if it is higher than 1mm), it will cause too much error in the subsequent calculation of the depth map, which will affect the accuracy of the subsequent conversion of camera coordinates to world coordinates, and even if there are other methods (such as adjusting the focal length of the second camera)
  • the lens center position of the second camera is close to the X axis (ideal position), but the effect is still limited, and the calculation of the depth map will also increase a considerable amount of calculation.
  • the camera configuration system mainly includes: a main camera group, a secondary camera group, and an arithmetic unit; wherein the main camera group can be at least It includes four main camera lenses, and each main camera lens can be respectively arranged on four reference lines that can form a rectangle; the auxiliary camera group can include at least four auxiliary camera lenses, and each main camera lens and each auxiliary camera lens can be arranged along The four reference lines are arranged in the same plane at intervals to form a rectangle; the arithmetic unit can be connected to each main camera lens and each sub camera lens respectively; the optical axis of each sub camera lens or each main camera lens is configured to be rotatable, Make the optical axes of the auxiliary camera lens and the main camera lens arranged on the reference line parallel to each other, and make the optical axes of the auxiliary camera lens and the main camera lens arranged on the reference line perpendicular to the corresponding reference line to complete the
  • the present invention can at least achieve the beneficial effect of obtaining unobstructed omnidirectional depth, and even if the number of cameras used is more than four, The cameras can still be arranged in a rectangular shape without being arranged in a polygonal shape, which increases the configuration cost of the camera system.
  • Figure 1 is a schematic diagram of conventional well-known camera calibration.
  • Fig. 2 is an architecture diagram of the camera configuration system of the present invention.
  • Figure 3 is a flow chart of the camera configuration of the present invention.
  • Figure 4 is a schematic diagram (1) of the implementation of the present invention.
  • Figure 5 is a schematic diagram (2) of the implementation of the present invention.
  • Figure 6 is a schematic diagram (3) of the implementation of the present invention.
  • Fig. 7 is another embodiment (1) of the present invention.
  • Fig. 8 is another embodiment (2) of the present invention.
  • Figure 2 is the architecture diagram of the camera configuration system of the present invention.
  • the present invention proposes an omnidirectional stereo vision camera configuration system 10, which includes a main camera group 101, a secondary camera group 102, and an arithmetic unit 103 ,among them:
  • the main camera group 101 includes at least four main camera lenses (1011-1014), and each main camera lens (1011-1014) can be respectively arranged on four base lines that can form a rectangle;
  • the auxiliary camera group 102 includes at least four auxiliary camera lenses (1021 ⁇ 1024), and each main camera lens (1011 ⁇ 1014) and each auxiliary camera lens (1021 ⁇ 1024) are arranged to be able to follow the four reference
  • the lines are arranged at intervals on the same plane as rectangles;
  • the arithmetic unit 103 is respectively connected to each main camera lens (1011-1014) and each sub-camera lens (1021-1024) in information connection, wherein the arithmetic unit 103 may have at least one processor (not shown in the figure, for example CPU, MCU), which is provided with the operation arithmetic unit 103, and has functions such as logic operations, temporary storage of the results of operations, storage of the position of execution instructions, and image processing;
  • processor not shown in the figure, for example CPU, MCU
  • each secondary camera lens (1021 ⁇ 1024) or each main camera lens (1011 ⁇ 1014) can be configured to be rotatable, so that the secondary camera lens (1021 ⁇ 1014) arranged on the reference line can be configured to be rotatable. 1024) and the lens optical axis of the main camera lens (1011 ⁇ 1014) are parallel to each other, and the lens optical axes of the secondary camera lens (1021 ⁇ 1024) and the main camera lens (1011 ⁇ 1014) arranged on the reference line correspond to each other
  • the reference line is vertical to complete camera calibration for both the main camera group 101 and the auxiliary camera group 102;
  • the extracted images can be used by the computing unit 103 to calculate at least two depth maps (also called parallax). Figure), and after the arithmetic unit 103 performs image fusion on each depth map, the occlusion area between the depth maps can be eliminated to generate a de-occlusion depth map;
  • the arithmetic unit 103 may perform an image stitching for each de-occlusion depth map calculated by each reference line to obtain an omnidirectional depth map;
  • each secondary camera lens (1021 ⁇ 1024) of the auxiliary camera group 102 or the main camera lens (1011 ⁇ 1014) of the main camera group 101 can be The main camera group 101 and the auxiliary camera group 102, which are configured to rotate to the left or right by a specific angle based on the optical axis of the lens, and are located on the same reference line, the optical axes of the lenses can respectively face the same direction before the camera calibration is completed Or different directions;
  • the degree of freedom can be divided into two types: translation and rotation.
  • the translation can include the X axis (front/rear) and the Y axis (left/right). ), Z axis (up/down), rotation can include pitch (Pitch), roll (Roll) and yaw (Yaw), but not limited to these degrees of freedom.
  • Figure 3 is the flow chart of the camera configuration of the present invention. Please also refer to " Figure 2", “ Figure 4" ⁇ " Figure 8".
  • the present invention proposes an omnidirectional stereo vision camera configuration method S ,include:
  • step S10 Arrange the camera group on the reference line (step S10): As shown in " Figure 4", arrange at least four main camera lenses (1011-1014) of a main camera group 101 on four references that can form a rectangle Line (L1 ⁇ L4), arrange at least four sub-camera lenses (1021 ⁇ 1024) of a sub-camera group 102 on each reference line (L1 ⁇ L4), so that each main camera lens (1011 ⁇ 1014) and each sub-camera The camera lenses (1021 ⁇ 1024) are arranged in a rectangle along the four reference lines (L1 ⁇ L4) at intervals on the same plane;
  • Step S20 Rotate at least one camera lens (1021 ⁇ 1024) of the auxiliary camera group 102, or rotate the lens optical axis of the main camera lens (1011 ⁇ 1014) of the main camera group 101 so that it is arranged in
  • the optical axes of the main imaging lens (1011 ⁇ 1014) and the auxiliary imaging lens (1021 ⁇ 1024) of the reference line (L1 ⁇ L4) are parallel to each other, and the main imaging lens (1011 ⁇ 1011 ⁇ 1014)
  • the lens optical axis of the secondary camera lens (1021 ⁇ 1024) is perpendicular to the corresponding reference line (L1 ⁇ L4) to complete the camera calibration.
  • these camera lenses (1011 ⁇ 1014, 1021 ⁇ 1024) can be The configuration is based on the optical axis of the lens, and the two parts are rotated to the left or right by a specific angle, as shown in “Figure 4". Please also refer to " Figure 2". The figure is located on the reference line L1.
  • the main camera lens 1011 can be rotated by ⁇ degrees to the left or right on the basis of its optical axis, and the auxiliary camera lenses (1021, 1022) located on the left and right sides of the main camera lens 1011 are due to the mirrors of the main camera lens 1011.
  • the centers are all arranged on the same reference line L1, so the optical axis of the sub-camera lens (1021, 1022) can be rotated by ⁇ degrees to allow the camera lenses (1011, 1021, 1022) on the same reference line L1 to shift the matrix
  • the y and z components (Ty, Tz) in T approach 0, so that the lens center coordinates of the camera lens (1011, 1021, 1022) form zero rotation with each other, and only the X-axis offset (Tx) is retained;
  • the main camera lens 1012 located on the reference line L2 can be rotated to the left or right by ⁇ degrees based on its optical axis, while the secondary camera lens (1022) located on the left and right sides of the main camera lens 1012 , 1023), since the lens centers of the main camera lens 1012 and the main camera lens 1012 are arranged on the same reference line L2, the optical axis of the sub camera lens (1022, 1023) can be rotated by ⁇ degrees to give way to the same reference line L2.
  • the y and z components (Ty, Tz) of the camera lens (1012, 1022, 1023) in the offset matrix T approach 0, so that the lens center coordinates of the camera lens (1012, 1022, 1023) form zero rotation with each other, and Only the X-axis offset (Tx) is retained.
  • the optical axis of the camera lens can be rotated by the rotation angle ⁇ and ⁇ respectively, which is the same as the aforementioned reference line L1, L2
  • the method is similar, and so on, so I won’t repeat it here.
  • 2 ⁇ +2 ⁇ +2 ⁇ +2 ⁇ can be ⁇ 360°;
  • step S30 the images extracted from the main camera group 101 and the auxiliary camera group 102 that are set on the same reference line and have completed camera calibration (for example, the captured images shown in " Figure 5", and This schematic diagram is only an example, and is not limited to the number of these images), an arithmetic unit 103 calculates at least two depth maps of different angles, and each depth map can be eliminated after the arithmetic unit 103 performs image fusion calculations. The occlusion area between the images is generated, and a de-occlusion depth map is generated. Please refer to "Figure 6".
  • the depth map on the left of the figure is the arithmetic unit 103 based on the main camera lens 101 and the sub camera lens 1021 located on the reference line L1.
  • the extracted image, and the generated depth map DL, and the depth map in the middle is the depth map DR generated by the arithmetic unit 103 based on the images extracted by the main camera lens 101 and the sub camera lens 1022 located at the reference line L1.
  • Both the depth map DL and the depth map DR in the middle can find part of the occlusion area
  • the depth map D on the right is the unoccluded depth map generated by the image fusion calculation of the depth map DL and the depth map DR;
  • step S40 The arithmetic unit 103 performs image stitching for each de-occlusion depth map calculated by each reference line to obtain an omnidirectional depth map.
  • FIG. 7 is another embodiment (1) of the present invention.
  • FIG. 2 This embodiment is similar to the technology of "FIG. 2" ⁇ “FIG. 6", and the main difference lies in ,
  • the quality of 3D information is related to the degree of texture of the objects in the image without relying on the active light source. For example, blankets and newspapers are objects with rich texture, while white paper and monochromatic walls are objects with no texture.
  • the light source is insufficient, such as at night or indoors without turning on the lights, the quality of the 3D information will also be affected.
  • the omnidirectional stereoscopic camera configuration system 10 of this embodiment may also include coupling A diffractive optical element 105 (DOE) in the computing unit 103, and the diffractive optical element 105 can be mounted on each main camera lens (1011-1024) of the main camera group 101, and the diffractive optical element 105 Mainly used to project light points on the surface of the object to assist in determining the three-dimensional depth of the subject and the shooting scene, that is, when step S30 is executed, if the ambient light source is insufficient or the texture feature of the subject is not obvious, it is coupled to the computing unit
  • the diffractive optical element 105 of 103 can project light points on the surface of the object to give texture and light source to the object, so as to generate the required pattern in a specific position or space to assist in judging the three-dimensional depth of the object and the shooting scene.
  • the omnidirectional stereoscopic camera configuration system 10 of this embodiment further includes a Lidar module 106 (Lidar) coupled to the computing unit 103 for measuring the transmitted and received pulse signals (such as pulse lasers)
  • Lidar Lidar
  • the format of the depth information can be, for example, a point cloud (Point Cloud, where the information can include horizontal angle, vertical angle, distance, intensity, line, id, and timestamp). (Laser Timestamp).
  • the arithmetic unit 103 can determine a time of flight (ToF) by measuring the time interval between the transmitted and received pulse signals by the optical module 106 coupled to it, and then calculate The depth information of the subject and the shooting scene, and before the lidar module 106 returns the depth information to the computing unit 103, the computing unit 103 may first perform the depth maps and the camera lenses generated during the execution of step S30
  • the extracted images (1011 ⁇ 1014, 1021 ⁇ 1024) are subjected to image segmentation for image segmentation for the arithmetic unit 103 to obtain the omnidirectional depth map, which can generate more accurate depths for objects or shooting scenes at a longer distance Information, to make up for the problem that if only the LiDAR module 106 is used to detect a longer distance subject or the depth of a shooting scene, the depth information returned will have the problem that the x and y information are not dense enough; hereby, this implementation
  • the omnidirectional stereo vision camera configuration system 10 of the example may also include a radar module 107
  • this embodiment uses the LiDAR module 106 and the radar module 107, It can solve the problem that when the present invention achieves omni-directional stereo vision through these camera lenses (1011-1014, 1021-1024), the depth information for objects or scenes at a longer distance may be insufficiently accurate.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Studio Devices (AREA)
  • Measurement Of Optical Distance (AREA)

Abstract

本发明提供了一种全向立体视觉的摄像机配置系统及摄像机配置方法，摄像机配置系统主要包括：至少四个主摄像镜头，分别排列于可构成一矩形的四基准线；至少四个副摄像镜头，各主摄像镜头与各副摄像镜头，可沿着四个基准线于同一平面间隔排列设置为矩形；本发明的关键在于，各副摄像镜头或各主摄像镜头的光轴，被配置为可分为两部分旋转，使排列于同一基准线的摄像镜头的光轴彼此平行，以及使排列于同一基准线的摄像镜头的光轴皆与基准线垂直，以完成摄像机校正；因此，可让对应于各基准线的至少两深度图于执行影像融合演算后，消除遮挡区域，进而能通过影像拼接获取无遮挡的全向深度图。

Description

全向立体视觉的摄像机配置系统及摄像机配置方法 技术领域
本发明涉及立体视觉(Stereo Vision)技术，尤指一种可将多个摄像镜头间隔排列设置为矩形，并使待校正的至少三个摄像镜头的镜心配置于同一基准线，以可因此获得无遮挡深度图，进而能于完成影像拼接(image stitching)后获取全向深度图(omnidirectional depth map)的「全向立体视觉的摄像机配置系统及摄像机配置方法」。
背景技术
在双目视觉中，需要对摄像机进行标定和校正，而摄像机校正(camera rectification)的目的是达成理想的双目摄像机系统，使得至少两个摄像机的光轴完全平行(即镜心彼此之间仅存在X分量)，并与基线垂直(左右摄像机的镜心的连线即为基线)，才能续行深度计算与三维重建。
一般而言，校正前左右摄像机的镜心(光心)并不是平行的，而校正后理想双目系统的左右摄像机的光轴彼此平行、光轴和影像平面垂直，且成像点在左右影像上的高度一致，依此，后续进行立体匹配(stereo matching)时，只需在同一列的搜索区间(search range)，搜索左右影像的匹配点即可，能使摄像机校正效率大大提高。
目前虽有人提出可利用3D检测摄像机(Stereoscopic Camera)取得全向深度信息，例如中国台湾发明专利第TW201832547号「产生全景深度影像的影像装置、相关方法及相关影像装置」所揭(下称台湾案)，台湾案主要使四颗超广角鱼眼镜头(>190度)两两背对与上下排列，并将提取的广角影像投影至经纬(equirectangular)坐标系，而位于同侧的左右两颗鱼眼镜头可用于计算180x180度的立体深度影像(stereo depth)，最后再将两侧的深度影像进行影像拼接，即可获取 360x180度的全向深度影像，然而，此种将广角影像投影至经纬坐标的等距柱状投影法(Equirectangular Projection)，在接近180度的影像解析度相当差(因使用鱼眼镜头会产生镜头扭曲现象)，同时，也会产生因采用立体视觉技术所产生的遮挡(Occlusion)问题，而遮挡问题将直接影响到深度估计的准确度。
虽另有发明人提出可将N颗摄像机排列成正N边形，并通过两两摄像机生成深度影像，最后再完成全景拼接以获取全向深度影像，即如美国发明专利第US10244226B2「Camera rig and stereoscopic image capture」所揭(下称美国案)，然而，此种作法虽可产生解析度较高的深度影像，但美国案摆放各摄像机的位置，将无法处理前述的遮挡问题，使得其产生的深度影像产生缺陷。
承上，为了在双目视觉领域获取无遮挡深度图(de-occlusion depth map)，多镜头(至少三颗)的摄像机校正是必要的，并请搭配参阅「图1」，假设图中所示的基线b12、b13分别为美国案校正前第一与第二摄像机、以及美国案校正前第一与第三摄像机的镜心O的连线，而基线b12’、b13’即为校正后第一摄像机与第二摄像机、以及校正后第一摄像机与第三摄像机的镜心O的连线，校正时，得以基线b13为基准线，故第一与第三摄像机只需通过旋转即可让X轴与基线b13的向量平行，至于第二摄像机虽也可通过偏移矩阵T(
Figure PCTCN2020128353-appb-000001
也可称平移矩阵)，达到让X轴与基线b12的向量平行，但由于第二摄像机偏离X轴太远，导致偏移矩阵T的y、z分量较大，而当y、z分量达到一定程度(例如若高于1mm)，将造成后续计算深度图的误差过大，进而影响到后续将摄像机坐标转换至世界坐标的准确性，且即便有其它方式(例如调整第二摄像机的焦距)可让第二摄像机的镜心位置逼近于X轴(理想位置)，但效果仍为有限，且届时在计算深度图时也会因此增加可观的运算量，反之，由于美国案的第一至第三摄像机的镜心并非排列于同一基线(尤其像第二摄像机偏离X轴太远)，故在无法取得第二摄像机所拍摄场景物体的XYZ信息(摄像机坐标)前，将无法计算出无遮挡视差图。
依此，如何提出一种可获取无遮挡视差图、毋须使用鱼眼镜头的全向立体视觉的摄像机配置系统及摄像机配置方法，乃有待解决的问 题。
发明内容
为达上述目的，本发明提出一种全向立体视觉的摄像机配置系统及摄像机配置方法，摄像机配置系统主要包括：一主摄像机组、一辅摄像机组及一运算单元；其中，主摄像机组可至少包括四个主摄像镜头，各主摄像镜头可分别排列于可构成一矩形的四个基准线；辅摄像机组可至少包括四个副摄像镜头，且各主摄像镜头与各副摄像镜头，可沿着四个基准线于同一平面间隔排列设置为矩形；运算单元可分别与各主摄像镜头及各副摄像镜头呈信息连结；各副摄像镜头或各主摄像镜头的光轴被配置为可旋转，使排列于基准线的副摄像镜头与主摄像镜头的光轴彼此平行，以及使排列于基准线的副摄像镜头与主摄像镜头的光轴皆与对应的基准线垂直，以完成摄像机校正；位于同一基准线且完成摄像机校正的该等摄像镜头，其所提取的多个影像可供运算单元演算出至少两深度图，且运算单元对各深度图执行影像融合演算后，可消除各深度图彼此之间的遮挡区域，而生成一去遮挡深度图；最后，运算单元可对于各基准线所演算出的各去遮挡深度图，进行一影像拼接而获取一全向深度图。
因此，本发明据以实施后，相较于既有获取全向深度图的作法，本发明至少可达成获取无遮挡的全向深度的有利功效，并且，即便采用的摄像机数量为四颗以上，仍可使该等摄像机维持矩形排列，而毋须排列为多边形而增加摄像机系统的配置成本。
为使贵审查委员得以清楚了解本发明的目的、技术特征及其实施后的功效，兹以下列说明搭配图示进行说明，敬请参阅。
附图说明
图1为常规熟知摄像机校正示意图。
图2本发明的摄像机配置系统架构图。
图3为本发明的摄像机配置流程图。
图4为本发明的的实施示意图(一)。
图5为本发明的的实施示意图(二)。
图6为本发明的的实施示意图(三)。
图7为本发明的另一实施例(一)。
图8为本发明的另一实施例(二)。
【符号说明】
O镜心                 b12基线
                      b12’基线
                      b13基线
                      b13’基线
10全向立体视觉的摄像机配置系统
101主摄像机组         101主摄像机组
                      1011主摄影镜头
                      1012主摄影镜头
                      1013主摄影镜头
                      1014主摄影镜头
102辅摄像机组         1021副摄影镜头
                      1022副摄影镜头
                      1023副摄影镜头
                      1024副摄影镜头
103运算单元
105绕射光学元件
106光达模组
107雷达模组
L1～L4基准线
D深度图               DL深度图
                      DR深度图
S全向立体视觉的摄像机配置方法
S10于基准线配置摄像机组
S20校正摄像机
S30产生深度信息
S40产生全向深度
具体实施方式
请参阅「图2」，其为本发明的摄像机配置系统架构图，本发明提出一种全向立体视觉的摄像机配置系统10，包括一主摄像机组101、一辅摄像机组102及一运算单元103，其中：
(1)主摄像机组101至少包括四个主摄像镜头(1011～1014)，各主摄像镜头(1011～1014)可分别排列于可构成一矩形的四个基准线(base line)；
(2)辅摄像机组102至少包括四个副摄像镜头(1021～1024)，且各主摄像镜头(1011～1014)与各副摄像镜头(1021～1024)，被配置为可沿着四个基准线于同一平面间隔排列设置为矩形；
(3)运算单元103分别与各主摄像镜头(1011～1014)及各副摄像镜头(1021～1024)呈信息连结，其中，运算单元103可至少具有一处理器(图中未绘示，例如CPU、MCU)，其供以运行运算单元103，并具备逻辑运算、暂存运算结果、保存执行指令位置、执行影像处理等功能；
(4)各副摄像镜头(1021～1024)或各主摄像镜头(1011～1014)的镜头光轴(Optical Axis)均可被配置为可旋转，使排列于基准线的副摄像镜头(1021～1024)与主摄像镜头(1011～1014)的镜头光轴彼此平行，以及使排列于基准线的副摄像镜头(1021～1024)与主摄像镜头(1011～1014)的镜头光轴皆与对应的基准线垂直，以对主摄像机组101与辅摄像机组102皆完成摄像机校正；
(5)位于同一基准线且完成摄像机校正的该等摄像镜头(1011～1014、1021～1024)，其所提取的影像可供运算单元103演算出至少两深度图(depth map，也可称视差图)，且运算单元103对各深度 图执行影像融合演算(fusion)后，可消除各深度图彼此之间的遮挡区域，而生成一去遮挡深度图；
(6)运算单元103可对于各基准线所演算出的各去遮挡深度图，进行一影像拼接而获取一全向深度图；
(7)承上，本发明在一较佳实施例中，辅摄像机组102的各副摄像镜头(1021～1024)，或是主摄像机组101的主摄像镜头(1011～1014)，系可被配置为以镜头光轴为基准向左或向右旋转一特定角度，且位于同一基准线的主摄像机组101与辅摄像机组102，其镜头光轴于完成摄像机校正前，系可分别朝向同一方向或不同方向；
(8)承上，本发明在一较佳实施例中，还包括耦接于运算单元103的一惯性传感器(IMU)，供以回传多个自由度(DOF)的运动信息与姿态信息，以更精确追踪被摄物体与拍摄场景如何在现实世界中运动，其中，自由度可例如区分为两种类型：平移和旋转，平移可包括X轴(前/后)、Y轴(左/右)、Z轴(上/下)，旋转可包括纵摇(Pitch)、横摇(Roll)和垂摇(Yaw)，但并不以此些自由度为限。
请参阅「图3」，其为本发明的摄像机配置流程图，并请搭配参阅「图2」、「图4」～「图8」，本发明提出一种全向立体视觉的摄像机配置方法S，包括：
(1)于基准线配置摄像机组(步骤S10)：如「图4」所示，将一主摄像机组101的至少四个主摄像镜头(1011～1014)配置于可构成一矩形的四个基准线(L1～L4)，将一辅摄像机组102的至少四个副摄像镜头(1021～1024)分别配置于各基准线(L1～L4)，使各主摄像镜头(1011～1014)与各副摄像镜头(1021～1024)沿着四个基准线(L1～L4)于同一平面间隔排列设置为矩形；
(2)校正摄像机(步骤S20)：旋转辅摄像机组102的至少一副摄像镜头(1021～1024)、或旋转主摄像机组101的主摄像镜头(1011～1014)的镜头光轴，使排列于基准线(L1～L4)的主摄像镜头(1011～1014)与副摄像镜头(1021～1024)的镜头光轴彼此平行，以及使排列于基准线(L1～L4)的主摄像镜头(1011～1014)与副摄像镜头(1021～1024)的镜头 光轴皆与对应的基准线(L1～L4)垂直，以完成摄像机校正，其中，该等摄像镜头(1011～1014、1021～1024)可被配置为以镜头光轴为基准，分别两部分向左或向右旋转一特定角度，即如「第4图」所示，并请搭配参阅「图2」，图中所示位于基准线L1的主摄像镜头1011，系可以其光轴为基准向左或向右旋转θ度，而位于主摄像镜头1011的左右两侧的副摄像镜头(1021、1022)，由于其与主摄像镜头1011的镜心皆排列于相同的基准线L1，故可将副摄像镜头(1021、1022)的光轴旋转θ度，以让位于相同基准线L1的摄像镜头(1011、1021、1022)在偏移矩阵T中的y、z分量(Ty、Tz)趋近于0，让摄像镜头(1011、1021、1022)的镜心坐标形成彼此零旋转，而只保留X轴向的偏移量(Tx)；
(3)承上，同样地，位于基准线L2的主摄像镜头1012，可以其光轴为基准向左或向右旋转Φ度，而位于主摄像镜头1012的左右两侧的副摄像镜头(1022、1023)，由于其与主摄像镜头1012的镜心皆排列于相同的基准线L2，故可将副摄像镜头(1022、1023)的光轴旋转Φ度，以让位于相同基准线L2的摄像镜头(1012、1022、1023)在偏移矩阵T中的y、z分量(Ty、Tz)趋近于0，让摄像镜头(1012、1022、1023)的镜心坐标形成彼此零旋转，而只保留X轴向的偏移量(Tx)，至于基准线L3与基准线L4的摄像机校正作法，则可分别以旋转角度φ、Ω旋转摄像镜头的光轴，与前述对于基准线L1、L2的作法类同，以此类推，于此不再赘述，其中，2θ+2Φ+2φ+2Ω可≥360°；
(4)承上，由于本发明于执行步骤S20时，并未改变各摄像镜头(1011～1014、1022～1024)的位置，故主摄像机组101与辅摄像机组102的摄像头所接收到的光线都是相同的，依此，可通过旋转镜心成功模拟镜头光轴所面对的方向；
(5)产生深度信息(步骤S30)：由设置于同一基准线、且完成摄像机校正的主摄像机组101与辅摄像机组102所提取的影像(例如「图5」所示的各拍摄影像，而本示意图仅为举例，并不以此些影像数量为限)，供一运算单元103演算出不同角度的至少两深度图，且各深 度图经运算单元103执行影像融合演算后，可消除各深度图彼此之间的遮挡区域，而生成一去遮挡深度图，请搭配参阅「图6」，图中左侧的深度图为运算单元103基于位于基准线L1的主摄像镜头101与副摄像镜头1021所提取的影像，而生成的深度图DL，而中间的深度图为运算单元103基于位于基准线L1的主摄像镜头101与副摄像镜头1022所提取的影像，而生成的深度图DR，从图中的深度图DL与深度图DR皆可发现部分的遮挡区域，而右侧的深度图D即为深度图DL与深度图DR作影像融合演算而生成的去遮挡深度图；
(6)产生全向深度图(步骤S40)：运算单元103对于各基准线所演算出的各去遮挡深度图，进行一影像拼接而获取一全向深度图。
请参阅「图7」，其为本发明的另一实施例(一)，并请搭配参阅「图2」，本实施例与「图2」～「图6」的技术类同，主要差异在于，由于在不靠投射主动光源的情况下，3D信息的品质与影像中物体的纹理程度有关，例如毛毯、报纸即为富含纹理的物体，而白纸、单色墙壁即为不具纹理的物体，另外若是光源不足的情况，例如夜晚或在室内但未开灯的情况下，也会影响3D信息的品质，依此，本实施例的全向立体视觉的摄像机配置系统10还可包含耦接于运算单元103的一绕射光学元件105(Diffractive Optical Element，DOE)，且绕射光学元件105可分别搭载于主摄像机组101的各个主摄像镜头(1011～1024)，而绕射光学元件105主要用于投射光点于物体表面，以辅助判断被摄物体与拍摄场景的三维深度，即于步骤S30执行时，若环境光源不足或被摄物体的纹理特征不明显时，耦接于运算单元103的绕射光学元件105可投射光点于物体表面，以给予被摄物体纹理与光源，达到于特定位置或空间产生所需的图案，以辅助判断被摄物体与拍摄场景的三维深度。
请参阅「图8」，其为本发明的另一实施例(二)，并请搭配参阅「图2」，本实施例与「图2」～「图7」的技术类同，主要差异在于，本实施例的全向立体视觉的摄像机配置系统10还包含耦接于运算单元103的一光达模组106(Lidar)，供以通过测量发送和接受到的脉冲 讯号(例如脉冲雷射)的时间间隔，来计算被摄物体与拍摄场景的深度信息，而深度信息的格式可例如为点云(Point Cloud，其中信息可包括水平角度、垂直角度、距离、强度、line、id、时间戳(Laser Timestamp)。意即，执行步骤S30时，运算单元103可通过与其耦接的光达模组106，测量发送和接受到的脉冲讯号的时间间隔来确定一飞行时间(ToF)，进而演算被摄物体与拍摄场景的深度信息，并且，在光达模组106尚未回传深度信息至运算单元103前，运算单元103可先对步骤S30执行时所产生的各深度图以及该等摄像镜头(1011～1014、1021～1024)所提取的影像进行影像分割(image segmentation)，以供运算单元103获取全向深度图时，可针对较远距离的被摄物体或拍摄场景生成较精确的深度信息，以弥补若仅采用光达模组106来侦测较远距离的被摄物体或拍摄场景的深度，其回传的深度信息会有x、y信息不够致密的问题；承上，本实施例的全向立体视觉的摄像机配置系统10还可包含耦接于运算单元103的一雷达模组107(Radar)，供以于步骤S30执行时接收空间内存在物体所反射的无线电波，以供运算单元103计算出被摄物体与拍摄场景的深度信息，且雷达模组107可例如为一毫米波雷达(mmWave Rader)，换言之，本实施例通过使用光达模组106与雷达模组107，可解决本发明通过该等摄像镜头(1011～1014、1021～1024)达成全向立体视觉时，针对较远距离的被摄物体或场景的深度信息可能会有不够准确的问题。
以上所述者，仅为本发明的较佳的实施例而已，并非用以限定本发明实施的范围；任何熟习此技艺者，在不脱离本发明的精神与范围下所作的均等变化与修饰，皆应涵盖于本发明的专利范围内。

Claims (10)

  1. 一种全向立体视觉的摄像机配置系统，包含：
    一主摄像机组，至少包括四个主摄像镜头，各所述主摄像镜头分别排列于可构成一矩形的四个基准线；
    一辅摄像机组，至少包括四个副摄像镜头，且各所述主摄像镜头与各所述副摄像镜头，沿着所述四个基准线于同一平面间隔排列设置为所述矩形；
    一运算单元，分别与各所述主摄像镜头及各所述副摄像镜头呈信息连结；
    各所述副摄像镜头或各所述主摄像镜头的光轴被配置为可旋转，使排列于所述基准线的所述副摄像镜头与所述主摄像镜头的光轴彼此平行，以及使排列于所述基准线的所述副摄像镜头与所述主摄像镜头的光轴皆与对应的所述基准线垂直，以完成摄像机校正；
    位于同一基准线且完成摄像机校正的所述主摄像镜头及所述副摄像镜头，其所提取的多个影像供所述运算单元演算出至少两深度图，且所述至少两深度图供所述运算单元执行影像融合演算，以消除各所述深度图彼此之间的遮挡区域，而生成一去遮挡深度图；以及
    所述运算单元对于各所述基准线所演算出的各所述去遮挡深度图，进行一影像拼接而获取一全向深度图。
  2. 如权利要求1所述的全向立体视觉的摄像机配置系统，其特征在于，还包含耦接于所述运算单元的一绕射光学元件，供以投射光点于物体表面，以辅助判断被摄物体与拍摄场景的三维深度。
  3. 如权利要求1所述的全向立体视觉的摄像机配置系统，其特征在于，所述辅摄像机组的各所述副摄像镜头，或所述主摄像机组的所述主摄像镜头，被配置为以光轴为基准向左或向右旋转。
  4. 如权利要求1或2所述的全向立体视觉的摄像机配置系统，其特征在于，还包括耦接于所述运算单元的一光达模组，供以通过测量发送和接受到的脉冲讯号的时间间隔，来计算被摄物体与拍摄场景的深度信息。
  5. 如权利要求1所述的全向立体视觉的摄像机配置系统，其特征在于，还包括耦接于所述运算单元的一惯性传感器，供以回传多个自由度的运动信息与姿态信息。
  6. 一种全向立体视觉的摄像机配置方法，包括：
    一于基准线配置摄像机组步骤：将一主摄像机组的至少四个主摄像镜头配置于可构成一矩形的四个基准线，将一辅摄像机组的至少四个副摄像镜头分别配置于各所述基准线，使各所述主摄像镜头与各所述副摄像镜头沿着所述四个基准线于同一平面间隔排列设置为所述矩形；
    一校正摄像机步骤：旋转所述辅摄像机组的至少一所述副摄像镜头的光轴，或旋转所述主摄像机组的所述主摄像镜头的光轴，使排列于所述基准线的所述主摄像镜头与所述副摄像镜头的光轴彼此平行，以及使排列于所述基准线的所述主摄像镜头与所述副摄像镜头的镜光轴皆与对应的所述基准线垂直，以完成摄像机校正；
    一产生深度信息步骤：由设置于同一基准线、且完成摄像机校正的所述主摄像机组与所述辅摄像机组所提取的影像，供一运算单元演算出至少两深度图，且各所述深度图经所述运算单元执行影像融合演算后，可消除各所述深度图彼此之间的遮挡区域，而生成一去遮挡深度图；以及
    一产生全向深度图步骤：所述运算单元对于各所述基准线所演算出的各所述去遮挡深度图，进行一影像拼接而获取一全向深度图。
  7. 如权利要求6所述的全向立体视觉的摄像机配置方法，其特征在于，所述产生深度信息步骤执行前，若光源不足或被摄物体的纹理特征不明显时，耦接于所述运算单元的一绕射光学元件投射光点于物体表面，并给予被摄物体纹理与光源，以辅助判断被摄物体与拍摄场景的三维深度。
  8. 如权利要求6所述的全向立体视觉的摄像机配置方法，其特征在于，所述产生深度信息步骤执行时，所述运算单元还通过与其耦接的一光达模组，测量发送和接受到的脉冲讯号的时间间隔来确定一 飞行时间，进而演算被摄物体与拍摄场景的深度信息。
  9. 如权利要求8所述的全向立体视觉的摄像机配置方法，其特征在于，所述产生深度信息步骤执行时，在所述光达模组尚未回传深度信息至所述运算单元前，所述运算单元先对各所述深度图以及所述主摄像镜头及所述副摄像镜头所提取的影像进行影像分割。
  10. 如权利要求6所述的全向立体视觉的摄像机配置方法，其特征在于，所述产生深度信息步骤执行时，耦接于所述运算单元的一雷达模组系接收空间内存在物体所反射的无线电波，以供所述运算单元计算出被摄物体与拍摄场景的深度信息。
PCT/CN2020/128353 2019-11-14 2020-11-12 全向立体视觉的摄像机配置系统及摄像机配置方法 WO2021093804A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201911114600.XA CN112804515A (zh) 2019-11-14 2019-11-14 全向立体视觉的摄像机配置系统及摄像机配置方法
CN201911114600.X 2019-11-14

Publications (1)

Publication Number Publication Date
WO2021093804A1 true WO2021093804A1 (zh) 2021-05-20

Family

ID=75804037

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/128353 WO2021093804A1 (zh) 2019-11-14 2020-11-12 全向立体视觉的摄像机配置系统及摄像机配置方法

Country Status (2)

Country Link
CN (1) CN112804515A (zh)
WO (1) WO2021093804A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115086629B (zh) * 2022-06-10 2024-02-27 谭健 球体多镜头实时全景三维成像系统

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN207407873U (zh) * 2017-11-26 2018-05-25 新疆工程学院 数字航空全景摄影测量装置
CN108322730A (zh) * 2018-03-09 2018-07-24 嘀拍信息科技南通有限公司 一种可采集360度场景结构的全景深度相机系统
CN108696739A (zh) * 2017-03-31 2018-10-23 钰立微电子股份有限公司 可校正遮蔽区的深度图产生装置
CN109660733A (zh) * 2019-01-04 2019-04-19 Oppo广东移动通信有限公司 电子设备和移动平台
US20190208181A1 (en) * 2016-06-10 2019-07-04 Lucid VR, Inc. Digital Camera Device for 3D Imaging

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
TW201612853A (en) * 2014-09-30 2016-04-01 Lite On Technology Corp Method for creating depth map and a multi-lens camera system using the method
EP3286914B1 (en) * 2015-04-19 2019-12-25 FotoNation Limited Multi-baseline camera array system architectures for depth augmentation in vr/ar applications
KR101673000B1 (ko) * 2015-11-11 2016-11-07 공간정보기술 주식회사 3안 스테레오 이미지를 이용한 피사체 윤곽선 개선 시스템
CN212163540U (zh) * 2019-11-14 2020-12-15 南京深视光点科技有限公司 全向立体视觉的摄像机配置系统

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190208181A1 (en) * 2016-06-10 2019-07-04 Lucid VR, Inc. Digital Camera Device for 3D Imaging
CN108696739A (zh) * 2017-03-31 2018-10-23 钰立微电子股份有限公司 可校正遮蔽区的深度图产生装置
CN207407873U (zh) * 2017-11-26 2018-05-25 新疆工程学院 数字航空全景摄影测量装置
CN108322730A (zh) * 2018-03-09 2018-07-24 嘀拍信息科技南通有限公司 一种可采集360度场景结构的全景深度相机系统
CN109660733A (zh) * 2019-01-04 2019-04-19 Oppo广东移动通信有限公司 电子设备和移动平台

Also Published As

Publication number Publication date
CN112804515A (zh) 2021-05-14

Similar Documents

Publication Publication Date Title
CN110044300B (zh) 基于激光器的两栖三维视觉探测装置及探测方法
JP7519126B2 (ja) アクティブアライメントを備えた拡張現実ディスプレイおよび対応する方法
US10085011B2 (en) Image calibrating, stitching and depth rebuilding method of a panoramic fish-eye camera and a system thereof
WO2019100933A1 (zh) 用于三维测量的方法、装置以及系统
US11145077B2 (en) Device and method for obtaining depth information from a scene
WO2018068719A1 (zh) 一种图像拼接方法及装置
JP4825980B2 (ja) 魚眼カメラの校正方法。
WO2018076154A1 (zh) 一种基于鱼眼摄像机空间位姿标定的全景视频生成方法
US20170127045A1 (en) Image calibrating, stitching and depth rebuilding method of a panoramic fish-eye camera and a system thereof
CN102243432A (zh) 全景立体摄像装置
WO2023201578A1 (zh) 单目激光散斑投影系统的外参数标定方法和装置
CN111854636B (zh) 一种多相机阵列三维检测系统和方法
WO2021104308A1 (zh) 全景深度测量方法、四目鱼眼相机及双目鱼眼相机
JP2010276433A (ja) 撮像装置、画像処理装置及び距離計測装置
CN107560554A (zh) 一种基于旋转透镜的三维信息视觉测量方法
JP4825971B2 (ja) 距離算出装置、距離算出方法、構造解析装置及び構造解析方法。
TWM594322U (zh) 全向立體視覺的相機配置系統
WO2021093804A1 (zh) 全向立体视觉的摄像机配置系统及摄像机配置方法
CN112950727B (zh) 基于仿生曲面复眼的大视场多目标同时测距方法
CN108205799B (zh) 一种图像拼接方法及装置
CN110766752B (zh) 一种带反光标志点的虚拟现实交互眼镜及空间定位方法
CN212163540U (zh) 全向立体视觉的摄像机配置系统
TWI725620B (zh) 全向立體視覺的相機配置系統及相機配置方法
CN116929290A (zh) 一种双目视角差立体深度测量方法、系统及存储介质
CN113034615B (zh) 一种用于多源数据融合的设备标定方法及相关装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20888633

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20888633

Country of ref document: EP

Kind code of ref document: A1