[go: up one dir, main page]
More Web Proxy on the site http://driver.im/

CN114862804A - Detection method and device, electronic equipment and storage medium - Google Patents

Detection method and device, electronic equipment and storage medium Download PDF

Info

Publication number
CN114862804A
CN114862804A CN202210531141.0A CN202210531141A CN114862804A CN 114862804 A CN114862804 A CN 114862804A CN 202210531141 A CN202210531141 A CN 202210531141A CN 114862804 A CN114862804 A CN 114862804A
Authority
CN
China
Prior art keywords
depth
value
pixel
confidence
current
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210531141.0A
Other languages
Chinese (zh)
Inventor
侯烨
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guangdong Oppo Mobile Telecommunications Corp Ltd
Original Assignee
Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guangdong Oppo Mobile Telecommunications Corp Ltd filed Critical Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority to CN202210531141.0A priority Critical patent/CN114862804A/en
Publication of CN114862804A publication Critical patent/CN114862804A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/0002Inspection of images, e.g. flaw detection
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00Image analysis
    • G06T7/50Depth or shape recovery
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10004Still image; Photographic image

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Quality & Reliability (AREA)
  • Image Analysis (AREA)

Abstract

In a detection method, a detection apparatus, an electronic device, and a non-volatile computer-readable storage medium according to the present application, the method includes: shooting a target scene to acquire a depth image, wherein the depth image comprises a plurality of depth pixels, and the depth value of each depth pixel has a corresponding confidence value; identifying the type of the target scene according to the confidence value; determining a confidence threshold according to the type of the target scene; and detecting whether the depth pixel is valid according to the confidence value and the confidence threshold value. The type of the target scene is determined through the distribution rule of the confidence value of the depth image obtained by shooting the target scene, if the target scene is a large object type, a large object large background type, a general type and the like, so that the confidence threshold value suitable for the current type is accurately determined according to the type of the target scene, whether the depth pixel is effective or not is accurately judged according to the accurate confidence threshold value, and the pixel point flicker phenomenon is reduced or even eliminated.

Description

检测方法及装置、电子设备及存储介质Detection method and device, electronic device and storage medium

技术领域technical field

本申请涉及检测技术领域,特别涉及一种检测方法、检测装置、电子设备及非易失性计算机可读存储介质。The present application relates to the field of detection technology, and in particular, to a detection method, a detection device, an electronic device, and a non-volatile computer-readable storage medium.

背景技术Background technique

目前,飞行时间(Time of flight,TOF)相机通过发射并接收调制光进行目标物体深度距离获取,随目标物体距离逐渐增大,接收到远距离信号光的强度与信噪比逐渐降低,计算得到的深度精度也逐渐变差甚至无法生成正确有效的深度距离。通常的做法是在置信度值低于某一阈值后,由算法将该位置的深度信息设置为无效,表征此时计算得到的深度距离精度过差或根本无法生成有效深度。At present, the time of flight (TOF) camera obtains the depth distance of the target object by transmitting and receiving modulated light. The depth accuracy of , also gradually deteriorates or even fails to generate correct and effective depth distances. The usual practice is to set the depth information of the position to be invalid by the algorithm after the confidence value is lower than a certain threshold, indicating that the depth distance calculated at this time has too poor accuracy or cannot generate a valid depth at all.

TOF相机与目标场景的不同局部的距离存在差异,存在某个局部对应的距离的置信度值在阈值附近,如电路噪声等原因使得激光的能量本身就不均匀,或者相机有微小的移动,不同时刻发射光的角度略有不同,接收端噪声等等,该局部的置信度值可能会在阈值附近波动,相应地,深度值就在有效和无效之间不停变化,出现像素点闪烁现象,影响成像效果。There are differences in the distance between the TOF camera and different parts of the target scene. The confidence value of the distance corresponding to a certain part is near the threshold. For example, the energy of the laser itself is not uniform due to circuit noise, or the camera moves slightly. The angle of the emitted light is slightly different at any time, the noise at the receiving end, etc., the local confidence value may fluctuate around the threshold value, and accordingly, the depth value is constantly changing between valid and invalid, and the phenomenon of pixel flicker occurs. affect the imaging effect.

发明内容SUMMARY OF THE INVENTION

本申请实施方式提供了一种检测方法、检测装置、电子设备及非易失性计算机可读存储介质。Embodiments of the present application provide a detection method, a detection apparatus, an electronic device, and a non-volatile computer-readable storage medium.

本申请实施例提供一种检测方法。所述检测方法包括:拍摄目标场景以获取深度图像,所述深度图像包括多个深度像素,每个所述深度像素的深度值均存在对应的置信度值,所述置信度值根据所述深度像素对应的光接收器接收的光量确定;根据所述置信度值识别所述目标场景的类型;根据所述目标场景的类型确定置信度阈值;及根据所述置信度值和所述置信度阈值检测所述深度像素是否有效。The embodiments of the present application provide a detection method. The detection method includes: photographing a target scene to obtain a depth image, the depth image includes a plurality of depth pixels, and the depth value of each of the depth pixels has a corresponding confidence value, and the confidence value is based on the depth. determining the amount of light received by the light receiver corresponding to the pixel; identifying the type of the target scene according to the confidence value; determining a confidence threshold according to the type of the target scene; and according to the confidence value and the confidence threshold Check if the depth pixel is valid.

本申请实施方式提供一种检测装置。所述检测装置包括拍摄模块、识别模块、第一确定模块和第二确定模块。所述拍摄模块用于拍摄目标场景以获取深度图像,所述深度图像包括多个深度像素,每个所述深度像素的深度值均存在对应的置信度值,所述置信度值根据所述深度像素对应的光接收器接收的光量确定。所述识别模块用于根据所述置信度值识别所述目标场景的类型。所述第一确定模块用于根据所述目标场景的类型确定置信度阈值。所述第二确定模块用于根据所述置信度值和所述置信度阈值检测所述深度像素是否有效。Embodiments of the present application provide a detection device. The detection device includes a photographing module, an identification module, a first determination module and a second determination module. The shooting module is used for shooting a target scene to obtain a depth image, the depth image includes a plurality of depth pixels, and the depth value of each of the depth pixels has a corresponding confidence value, and the confidence value is based on the depth. The amount of light received by the light receiver corresponding to the pixel is determined. The identifying module is configured to identify the type of the target scene according to the confidence value. The first determining module is configured to determine a confidence threshold according to the type of the target scene. The second determining module is configured to detect whether the depth pixel is valid according to the confidence value and the confidence threshold.

本申请实施方式提供一种电子设备。所述电子设备包括处理器,所述处理器用于执行检测方法。所述检测方法包括:拍摄目标场景以获取深度图像,所述深度图像包括多个深度像素,每个所述深度像素的深度值均存在对应的置信度值,所述置信度值根据所述深度像素对应的光接收器接收的光量确定;根据所述置信度值识别所述目标场景的类型;根据所述目标场景的类型确定置信度阈值;及根据所述置信度值和所述置信度阈值检测所述深度像素是否有效。Embodiments of the present application provide an electronic device. The electronic device includes a processor for performing the detection method. The detection method includes: photographing a target scene to obtain a depth image, the depth image includes a plurality of depth pixels, and the depth value of each of the depth pixels has a corresponding confidence value, and the confidence value is based on the depth. determining the amount of light received by the light receiver corresponding to the pixel; identifying the type of the target scene according to the confidence value; determining a confidence threshold according to the type of the target scene; and according to the confidence value and the confidence threshold Check if the depth pixel is valid.

本申请实施方式提供一种非易失性计算机可读存储介质,其上存储有计算机程序。该计算机程序被处理器执行时实现检测方法。所述检测方法包括:拍摄目标场景以获取深度图像,所述深度图像包括多个深度像素,每个所述深度像素的深度值均存在对应的置信度值,所述置信度值根据所述深度像素对应的光接收器接收的光量确定;根据所述置信度值识别所述目标场景的类型;根据所述目标场景的类型确定置信度阈值;及根据所述置信度值和所述置信度阈值检测所述深度像素是否有效。Embodiments of the present application provide a non-volatile computer-readable storage medium on which a computer program is stored. The computer program implements the detection method when executed by the processor. The detection method includes: photographing a target scene to obtain a depth image, the depth image includes a plurality of depth pixels, and the depth value of each of the depth pixels has a corresponding confidence value, and the confidence value is based on the depth. determining the amount of light received by the light receiver corresponding to the pixel; identifying the type of the target scene according to the confidence value; determining a confidence threshold according to the type of the target scene; and according to the confidence value and the confidence threshold Check if the depth pixel is valid.

本申请中检测方法、检测装置、电子设备及非易失性计算机可读存储介质中,通过拍摄目标场景获取的深度图像的置信度值的分布规律,来确定目标场景的类型,如目标场景为大物体类型、大物体大背景类型、通用类型等,从而根据目标场景的类型,准确地确定适应当前类型的置信度阈值,并根据较为准确地置信度阈值来准确地判断深度像素是否有效,从而降低甚至消除像素点闪烁现象。In the detection method, detection device, electronic device and non-volatile computer-readable storage medium in the present application, the type of the target scene is determined by the distribution law of the confidence value of the depth image obtained by shooting the target scene. For example, the target scene is Large object type, large object background type, general type, etc., so as to accurately determine the confidence threshold suitable for the current type according to the type of target scene, and accurately judge whether the depth pixel is valid according to the more accurate confidence threshold, so as to Reduce or even eliminate pixel flickering.

本申请实施方式的附加方面和优点将在下面的描述中部分给出,部分将从下面的描述中变得明显,或通过本申请的实践了解到。Additional aspects and advantages of embodiments of the present application will be set forth, in part, in the following description, and in part will be apparent from the following description, or learned by practice of the present application.

附图说明Description of drawings

本申请的上述和/或附加的方面和优点可以从结合下面附图对实施方式的描述中将变得明显和容易理解,其中:The above and/or additional aspects and advantages of the present application will become apparent and readily understood from the following description of embodiments in conjunction with the accompanying drawings, wherein:

图1是本申请某些实施方式的检测方法的流程示意图;Fig. 1 is the schematic flow chart of the detection method of some embodiments of the present application;

图2是本申请某些实施方式的检测方法的原理示意图;Fig. 2 is the principle schematic diagram of the detection method of some embodiments of the present application;

图3是本申请某些实施方式的检测方法的原理示意图;Fig. 3 is the principle schematic diagram of the detection method of some embodiments of the present application;

图4是本申请某些实施方式的检测方法的原理示意图;Fig. 4 is the principle schematic diagram of the detection method of some embodiments of the present application;

图5是本申请某些实施方式的检测方法的流程示意图;5 is a schematic flowchart of the detection method of some embodiments of the present application;

图6是本申请某些实施方式的检测方法的流程示意图;6 is a schematic flowchart of the detection method of some embodiments of the present application;

图7是本申请某些实施方式的检测方法的流程示意图;7 is a schematic flowchart of a detection method according to some embodiments of the present application;

图8是本申请某些实施方式的检测方法的流程示意图;8 is a schematic flowchart of a detection method according to some embodiments of the present application;

图9是本申请某些实施方式的检测方法的流程示意图;9 is a schematic flowchart of a detection method according to some embodiments of the present application;

图10是本申请某些实施方式的检测装置的模块示意图;10 is a schematic block diagram of a detection device according to some embodiments of the present application;

图11是本申请某些实施方式的电子设备的平面示意图;及11 is a schematic plan view of an electronic device according to certain embodiments of the present application; and

图12是本申请某些实施方式的非易失性计算机可读存储介质与处理器的交互示意图。FIG. 12 is a schematic diagram of interaction between a non-volatile computer-readable storage medium and a processor according to some embodiments of the present application.

具体实施方式Detailed ways

下面详细描述本申请的实施方式,实施方式的示例在附图中示出,其中,相同或类似的标号自始至终表示相同或类似的元件或具有相同或类似功能的元件。下面通过参考附图描述的实施方式是示例性的,仅用于解释本申请的实施方式,而不能理解为对本申请的实施方式的限制。Embodiments of the present application are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary, only used to explain the embodiments of the present application, and should not be construed as limitations on the embodiments of the present application.

下面首先对本申请出现的名词进行解释:First, the terms appearing in this application are explained below:

TOF成像:散斑TOF相机,即TX端(即光发射器端)发射的调制红外光是若干束一束一束的光束,而不是泛光(面光),因此经场景目标物体反射后,RX端(即接收器端)上接收到的是若干光斑,即深度图上被光斑照亮的区域才会存在有效的深度值,剩余区域都是无效值,若剩余这些区域出现了有效的深度值,则这区域出现的深度值像素点为干扰点,需要去除。当然,每个散斑可能同时打在几个像素点上,但是经过算法处理后,这几个像素点通常只生成一个深度值,即一个光斑点只生成一个像素点。TOF imaging: Speckle TOF camera, that is, the modulated infrared light emitted by the TX end (ie the light transmitter end) is a beam of beams, not flood light (surface light), so after being reflected by the target object in the scene, The RX end (ie the receiver end) receives a number of light spots, that is, only the areas illuminated by the light spots on the depth map have valid depth values, and the remaining areas are invalid values. If the remaining areas have valid depths value, the depth value pixels in this area are interference points and need to be removed. Of course, each speckle may hit several pixels at the same time, but after algorithm processing, these pixels usually only generate one depth value, that is, one light spot only generates one pixel.

在散斑TOF相机设计好之后,通常散斑在图像传感器上的位置比较固定。当然,光斑也会随着探测距离的远近发生几个像素的偏移,但这些偏移通常在一个小范围内,大体位置是比较固定的。因此,并不能将记录每个散斑出现位置的表格作为判断该区域出现的散斑点是否为干扰点,尤其是散斑点较密集的情况。After the speckle TOF camera is designed, the position of the speckle on the image sensor is usually relatively fixed. Of course, the light spot will also shift by several pixels with the detection distance, but these shifts are usually within a small range, and the general position is relatively fixed. Therefore, the table recording the occurrence position of each speckle cannot be used to judge whether the speckle appearing in this area is an interference point, especially when the speckle is relatively dense.

相比于散斑TOF相机,泛光TOF相机TX端发射的是面光源,理论上泛光TOF相机的图像传感器的每个像素点都可以接收到信号光,即每个像素点都可以解算出一个深度距离,即可以生成一张稠密的深度图,记录场景每个点与TOF相机之间的距离。Compared with the speckle TOF camera, the TX end of the flood TOF camera emits a surface light source. In theory, each pixel of the image sensor of the flood TOF camera can receive signal light, that is, each pixel can be calculated. A depth distance, that is, a dense depth map can be generated to record the distance between each point of the scene and the TOF camera.

TOF相机包括直接测量飞行时间(Direct TOF,dTOF)和间接测量飞行时间(Indirect TOF,iTOF)。dTOF会在单帧测量时间内发射和接收N次光信号,然后对记录的N次飞行时间做直方图统计,其中出现频率最高的飞行时间用于计算目标距离。iTOF采用了一种测相位偏移的方法,即发射正弦波/方波与接收正弦波/方波之间相位差,来确定目标距离。TOF cameras include direct measurement of time of flight (Direct TOF, dTOF) and indirect measurement of time of flight (Indirect TOF, iTOF). dTOF will transmit and receive N times of optical signals within a single frame measurement time, and then make histogram statistics on the recorded N times of flight time, among which the flight time with the highest frequency is used to calculate the target distance. iTOF adopts a method of measuring phase offset, that is, the phase difference between the transmitted sine/square wave and the received sine/square wave, to determine the target distance.

对iTOF或者dTOF来讲,虽然两者测距、成像原理不同,但都会同时生成一张深度图(Depth)和一张光强图/置信度图(Confidence)。Depth是一张二维的图像,图像中每个点(每个像素点)的值都是一个距离,表征TOF相机与真实三维场景待测点之间的垂直距离。TX端发射经过调试的红外光,经场景中目标物体反射后被RX端接收,后经其他硬件电路与软件算法对RX端接收到的调制信号光进行解算,生成二维的深度图。因此,RX端接收到的信号光越弱,则解算得到的距离深度值可信度通常越差,Depth的每个深度像素均在Confidence中存在对应的置信度值。For iTOF or dTOF, although the ranging and imaging principles of the two are different, a depth map (Depth) and a light intensity map/confidence map (Confidence) will be generated at the same time. Depth is a two-dimensional image, and the value of each point (each pixel) in the image is a distance, which represents the vertical distance between the TOF camera and the point to be measured in the real three-dimensional scene. The TX end emits the debugged infrared light, which is reflected by the target object in the scene and received by the RX end, and then the modulated signal light received by the RX end is solved by other hardware circuits and software algorithms to generate a two-dimensional depth map. Therefore, the weaker the signal light received at the RX end is, the lower the reliability of the calculated distance depth value is generally, and each depth pixel of Depth has a corresponding confidence value in Confidence.

当然,也并不是RX端接收到的光越强,深度值的置信度就越高,当光强大于某一值时,生成的深度距离值都是可信的,但当光强大到超过一定范围时,会产生过曝、近距离反射光太强会影响到邻近远距离区域深度值的解算。因此,通常将RX端每个深度像素接收到的光强值表征该深度像素的深度值的置信度。当然,对iTOF和dTOF生成Confidence的方法和原理也不同,不同厂家iTOF生成Confidence的方案也会有细小差别,有些设计会剔除环境中相同波段干扰光,有些则不会,这些都不影响本方案的阐述、这里不做详细讨论。Of course, it is not that the stronger the light received by the RX end, the higher the confidence of the depth value. When the light intensity is greater than a certain value, the generated depth distance value is credible, but when the light intensity exceeds a certain value When the range is too large, overexposure will occur, and the reflection light at close range will be too strong, which will affect the calculation of the depth value of the adjacent distant area. Therefore, the light intensity value received by each depth pixel at the RX end usually represents the confidence of the depth value of the depth pixel. Of course, the methods and principles of generating Confidence for iTOF and dTOF are also different, and the solutions for generating Confidence from iTOF of different manufacturers will also have subtle differences. Some designs will eliminate the same band of interference light in the environment, and some will not. These do not affect this solution. , and will not be discussed in detail here.

请参阅图1,本申请实施方式的检测方法包括:Please refer to FIG. 1, the detection method of the embodiment of the present application includes:

步骤011:拍摄目标场景以获取深度图像,深度图像包括多个深度像素,每个深度像素的深度值均存在对应的置信度值,置信度值根据深度像素对应的光接收器接收的光量确定。Step 011: Shoot the target scene to obtain a depth image, the depth image includes a plurality of depth pixels, the depth value of each depth pixel has a corresponding confidence value, and the confidence value is determined according to the amount of light received by the light receiver corresponding to the depth pixel.

具体地,TOF相机可获取目标场景的深度图像,深度图像包括多个深度像素,每个深度像素的深度值均在TOF相机根据光接收器接收的光量生成的置信度图像中存在对应的置信度值,即深度值和置信度值一一对应。TOF相机的RX端包括多个光接收器,如深度像素和光接收器一一对应,深度像素的置信度值根据对应的光接收器接收的光量来确定。Specifically, the TOF camera can obtain a depth image of the target scene, the depth image includes a plurality of depth pixels, and the depth value of each depth pixel has a corresponding confidence level in the confidence level image generated by the TOF camera according to the light amount received by the light receiver value, that is, the depth value and the confidence value have a one-to-one correspondence. The RX end of the TOF camera includes multiple light receivers, such as depth pixels and light receivers in one-to-one correspondence, and the confidence value of the depth pixels is determined according to the amount of light received by the corresponding light receiver.

可选地,可根据预设标定信息和深度像素的图像坐标,修正置信度值。Optionally, the confidence value may be corrected according to preset calibration information and image coordinates of depth pixels.

由于视角原因,图像传感器(sensor)上不同位置的像素接收到信号光的光程是不一样(即不同的深度像素对应的光程不同)。假设目标场景是一个大的平面,则目标场景的中心垂直距离光程最短,四周边角光程最长,光程越长,光线损耗越大,因此对于同样的距离,图像传感器上不同位置像素点的置信度值就会不一样,此时需要先对置信度值进行校正,以使得每个置信度值均为垂直距离的光程对应的置信度值。如(式2.1)所示为置信度的校正公式:Due to the viewing angle, the optical paths of the signal light received by the pixels at different positions on the image sensor (sensor) are different (that is, the optical paths corresponding to different depth pixels are different). Assuming that the target scene is a large plane, the vertical distance from the center of the target scene is the shortest, and the four peripheral corners have the longest optical path. The longer the optical path, the greater the light loss. Therefore, for the same distance, pixels at different positions on the image sensor The confidence value of the point will be different. At this time, the confidence value needs to be corrected first, so that each confidence value is the confidence value corresponding to the optical path of the vertical distance. As shown in (Equation 2.1), the correction formula for confidence is:

C(x,y)=α·Ccenter (式2.1)C (x,y) = α·C center (Equation 2.1)

其中C(x,y)为sensor不同位置像素点的置信度值,Ccenter为sensor中心位置像素点的置信度值,α为四周不同位置与中心位置接收到光的衰减系数,α与发射光的强度、反射光视场角、TX光场分布等因素相关,图像中不同位置的α值是不同的,α可以是预先设置好的也可以是通过标定得到的预设标定信息确定的。在TX端光场分布均匀的假设下,Confidence上各点的α值与该点在图像上所处的位置相关,即与该点接收反射光线的视场角相关,此时可以通过理论计算将二维平面上各点的α值计算出来。在实际情况下,视场中心的光程最短,因此置信度值也最大,解算得到的深度值的置信度值和精度均较高,反之,四周深度值的置信度值与精度都比较低,所以在TOF相机硬件设计时,可能会设计非均匀的TX端光源,加强四周的发射光强。基于此,通过标定的α值更加具有实际意义,准确性也较高。Among them, C(x,y) is the confidence value of the pixel points at different positions of the sensor, Ccenter is the confidence value of the pixel point at the center position of the sensor, α is the attenuation coefficient of the light received at different positions around the center and the center position, α and the emitted light Factors such as intensity, reflected light field angle, and TX light field distribution are related. The α value at different positions in the image is different. α can be preset or determined by preset calibration information obtained through calibration. Under the assumption that the light field distribution at the TX end is uniform, the α value of each point on the Confidence is related to the position of the point on the image, that is, related to the field angle of the reflected light received by the point. At this time, the theoretical calculation can be used to calculate The alpha value of each point on the two-dimensional plane is calculated. In the actual situation, the optical path in the center of the field of view is the shortest, so the confidence value is also the largest, and the confidence value and accuracy of the depth value obtained by the solution are both high. On the contrary, the confidence value and accuracy of the surrounding depth values are relatively low. , so in the hardware design of the TOF camera, a non-uniform TX end light source may be designed to enhance the emitted light intensity around. Based on this, the calibrated α value has more practical significance and higher accuracy.

步骤012:根据置信度值识别目标场景的类型。Step 012: Identify the type of the target scene according to the confidence value.

具体地,在确定每个深度像素的深度值和置信度值后,可根据所有深度像素的置信度值先生成置信度值直方图;然后根据置信度值直方图识别目标场景的类型。如可以根据深度图像的置信度值的分布,来识别目标场景的类型。Specifically, after determining the depth value and confidence value of each depth pixel, a confidence value histogram can be generated according to the confidence value of all depth pixels; then the type of the target scene can be identified according to the confidence value histogram. For example, the type of the target scene can be identified according to the distribution of the confidence value of the depth image.

需要提到的是,上述置信度值直方图所有值是针对泛光TOF相机而言的,散斑TOF相机由于图像中并不是所有点都是有效值,应当均修正为有效值的置信度值直方图。It should be mentioned that all the values of the above confidence value histogram are for the flood TOF camera. Since not all points in the image of the speckle TOF camera are valid values, they should be corrected to the confidence values of the valid values. histogram.

步骤013:根据目标场景的类型确定置信度阈值。Step 013: Determine a confidence threshold according to the type of the target scene.

具体地,目标场景的类型可以包括多种不同的类型,可根据实际拍摄场景的不同设置不同的类型。下面以三种典型的场景类型为例进行说明,但并不意味着目标场景的类型仅包括以下三种举例说明的类型。Specifically, the types of the target scene may include multiple different types, and different types may be set according to different actual shooting scenes. The following takes three typical scene types as examples for description, but it does not mean that the types of target scenes only include the following three types of examples.

场景类型一:大物体大背景类型Scene type 1: large object and large background type

对当前场景采集到的Confidence(即深度图像对应的置信度图像)做直方图统计,请参阅图2,为横纵轴分别为置信度值和数量的直方图,若Confidence直方图出现明显的双峰,即,某两个置信度值附近的像素点数量最多(如图2中的z5和z9的数量最多),则该场景可能一个是大的目标物体和一个大的背景,此时可能更关心目标物体的深度值,背景的深度值可有可无,则可以将置信度阈值设置为两峰之间谷值,如z8对应的置信度值范围的最大值或最小值;或者,z8对应的所有置信度值中的最大值或最小值;或者,z9对应的置信度值范围的最小值,或者z9对应的所有置信度值中的最小值。即仅保留目标物体对应的像素点,把背景的置信度值(如置信度值小于或等于z8对应的置信度值范围的最大值的深度像素均作为为背景)对应的像素点全部置为无效,可以防止背景区域的像素点闪烁问题。Do histogram statistics on the Confidence (that is, the confidence image corresponding to the depth image) collected in the current scene, please refer to Figure 2, which is a histogram whose horizontal and vertical axes are the confidence value and quantity respectively. Peak, that is, the number of pixels near a certain two confidence values is the largest (the number of z5 and z9 in Figure 2 is the largest), then the scene may be a large target object and a large background. If you care about the depth value of the target object, and the depth value of the background is optional, you can set the confidence threshold to the valley value between the two peaks, such as the maximum or minimum value of the confidence value range corresponding to z8; or, the corresponding value of z8 The maximum or minimum value among all confidence values; or, the minimum value of the range of confidence values corresponding to z9, or the minimum value among all confidence values corresponding to z9. That is, only the pixels corresponding to the target object are kept, and all the pixels corresponding to the confidence value of the background (for example, the depth pixels whose confidence value is less than or equal to the maximum value of the confidence value range corresponding to z8 are regarded as the background) are all invalid. , which can prevent pixel flickering in the background area.

场景类型二:大物体类型Scene Type 2: Large Object Type

对当前场景采集到的Confidence做直方图统计,请参阅图3,为横纵轴分别为置信度值和数量的直方图,若Confidence直方图出现明显的单峰集中(如图3中的z9),则该场景可能是只有一个大的目标物体,其余都是超出量程远距离或者反射率很低的物体,此时把置信度阈值设置为该单峰较小端的峰底的置信度值,如z9对应的置信度值范围的最小值,或者z9对应的所有置信度值中的最小值。Do histogram statistics for the Confidence collected in the current scene, please refer to Figure 3, which is a histogram whose horizontal and vertical axes are the confidence value and quantity respectively. If the Confidence histogram has obvious single peak concentration (z9 in Figure 3) , then the scene may have only one large target object, and the rest are objects that are far beyond the range or have low reflectivity. In this case, set the confidence threshold to the confidence value of the peak bottom at the smaller end of the single peak, such as The minimum value of the range of confidence values corresponding to z9, or the minimum value among all confidence values corresponding to z9.

场景类型三:通用类型Scenario Type 3: General Type

上述两种决策其实不适用于一般场景,只适应于某些特定场景。The above two decisions are actually not applicable to general scenarios, but only to certain specific scenarios.

请参阅图4,对于较为一般的通用场景,Confidence直方图可能没有明显的峰谷,这种场景在实际中更为常见。此时考虑Confidence直方图的分布,获取置信度值位于预设置信度值范围内的深度像素的数量占所有深度像素的数量的比例,然后根据该比例来确定目标场景的类型。Please refer to Figure 4. For a more general general scenario, the Confidence histogram may not have obvious peaks and valleys, which is more common in practice. At this time, considering the distribution of the Confidence histogram, obtain the ratio of the number of depth pixels whose confidence value is within the range of the preset confidence value to the number of all depth pixels, and then determine the type of the target scene according to the ratio.

例如,Confidence直方图所有值(或某一个比例,比如95%的点)均大于阈值C1(经验值,TOF相机制作完成之后进行测试,置信度值大于该值,深度值都是准确的),则把置信度阈值设为C1,此时目标场景类型可能绝大多数都是近距离物体。或者,不设置置信度阈值,即所有深度值均是有效的。For example, all the values of the Confidence histogram (or a certain proportion, such as 95% of the points) are greater than the threshold C1 (the empirical value, the TOF camera is tested after the production is completed, the confidence value is greater than this value, the depth value is accurate), Then set the confidence threshold to C1, at this time, most of the target scene types may be close-range objects. Alternatively, no confidence threshold is set, i.e. all depth values are valid.

再例如,Confidence直方图一定比例点的值(比如20%、30%的点)均小于阈值C2(经验值,TOF相机制作完成之后进行测试,在目标场景常见反射率下,比如50%-90%,置信度值小于该值,深度值则对应设计量程的中远距离),此时置信度阈值应当设置小于C2,但应当大于阈值C3,以将场景中包含的置信度值较低而更容易闪烁的中远距离的物体或背景的像素设置为无效。这里的C3是通用场景类型下的测试值或者经验值,置信度值小于C3,则对应的深度值大概率是不准确的,或者误差较大的,应当被去掉。For another example, the value of a certain proportion of points in the Confidence histogram (such as 20% and 30% points) is less than the threshold C2 (experience value, test after the TOF camera is completed, under the common reflectivity of the target scene, such as 50%-90 %, the confidence value is less than this value, and the depth value corresponds to the mid-to-long distance of the design range), at this time, the confidence threshold should be set less than C2, but should be greater than the threshold C3, so that the confidence value contained in the scene is lower and easier Blinking medium and long distance objects or background pixels are set to invalid. Here C3 is the test value or empirical value under the general scene type. If the confidence value is less than C3, the corresponding depth value is likely to be inaccurate, or the error is large, and should be removed.

可选地,在不同反射率下,C3(或C2)的值均是不一样的,可根据深度像素的反射率对应的预设阈值(如阈值C3)和目标场景的类型确定每个深度像素对应的置信度阈值。Optionally, under different reflectivity, the value of C3 (or C2) is different, and each depth pixel can be determined according to the preset threshold (such as threshold C3) corresponding to the reflectivity of the depth pixel and the type of the target scene. The corresponding confidence threshold.

其中,每个深度像素的反射率可根据深度像素的深度值和置信度值确定,再对应查找预设的反射率和阈值C3(或C2)的映射表,针对不同的深度像素,可使用不同的阈值,以提高置信度阈值的设置准确性。反射率计算公式如(式2.2)所示:Among them, the reflectivity of each depth pixel can be determined according to the depth value and confidence value of the depth pixel, and then the preset reflectivity and threshold C3 (or C2) mapping table can be searched correspondingly. For different depth pixels, different depth pixels can be used. to improve the accuracy of setting the confidence threshold. The calculation formula of reflectance is shown in (Equation 2.2):

Figure BDA0003646279110000041
Figure BDA0003646279110000041

(式2.2)中r_((x,y))为图像坐标为(x,y)的深度像素的反射率,α_(x,y)是(式2.1)中图像坐标为(x,y)的深度像素对应的α。如在通用场景类型下,根据通用场景类型及当前深度像素的反射率对应的阈值C3和阈值C2来确定置信度阈值,使得置信度阈值位于C2和C3之间。In (Equation 2.2), r_((x, y)) is the reflectivity of the depth pixel whose image coordinate is (x, y), and α_(x, y) is the image coordinate (x, y) in (Equation 2.1). The alpha corresponding to the depth pixel. For example, in the general scene type, the confidence threshold is determined according to the general scene type and the threshold C3 and the threshold C2 corresponding to the reflectivity of the current depth pixel, so that the confidence threshold is located between C2 and C3.

可选地,反射率计算公式还可以是如(式2.3)所示:Optionally, the reflectance calculation formula can also be as shown in (Equation 2.3):

r(x,y)=confidence(x,y)·(depth(x,y)(x,y))2 (式2.3)r (x,y) =confidence (x,y) ·(depth (x,y)(x,y) ) 2 (Equation 2.3)

(式2.3)中r_((x,y))为图像坐标为(x,y)的深度像素的反射率,(depth(x,y)(x,y))是图像坐标为(x,y)的深度像素对应的目标物体相对TOF相机的实际光程,其中,depth(x,y)为图像坐标为(x,y)的深度像素的深度值,β(x,y)可结合TOF相机的内参确定。In (Equation 2.3), r_((x,y)) is the reflectivity of the depth pixel whose image coordinate is (x, y), and (depth (x,y)(x,y) ) is the image coordinate whose (x,y) , the actual optical path of the target object relative to the TOF camera corresponding to the depth pixel of y), where depth (x, y) is the depth value of the depth pixel whose image coordinates are (x, y), and β (x, y) can be combined with The internal parameters of the TOF camera are determined.

如在通用场景类型下,根据通用场景类型及当前深度像素的反射率对应的阈值C3和阈值C2来确定置信度阈值,使得置信度阈值位于C2和C3之间。For example, in the general scene type, the confidence threshold is determined according to the general scene type and the threshold C3 and the threshold C2 corresponding to the reflectivity of the current depth pixel, so that the confidence threshold is located between C2 and C3.

此时又会有两种策略,完整性优先还是准确率优先,完整性优先,则置信度阈值应当靠近C3,准确率优先,则置信度阈值应当靠近C2。At this time, there will be two strategies, integrity priority or accuracy priority, integrity priority, then the confidence threshold should be close to C3, accuracy priority, then the confidence threshold should be close to C2.

步骤014:根据置信度值和置信度阈值检测深度像素是否有效。Step 014: Detect whether the depth pixel is valid according to the confidence value and the confidence threshold.

在确定了置信度阈值后,判断每个深度像素对应的置信度阈值是否小于置信度阈值,若小于置信度阈值,则可确定深度像素无效,深度像素有效即表示深度像素的深度值为有效深度值,深度像素无效即表示深度像素的深度值为无效深度值,有效深度值才会在后续进行三维重建等应用中使用。After the confidence threshold is determined, it is determined whether the confidence threshold corresponding to each depth pixel is less than the confidence threshold. If it is less than the confidence threshold, it can be determined that the depth pixel is invalid. The valid depth pixel means that the depth value of the depth pixel is a valid depth. If the depth pixel is invalid, it means that the depth value of the depth pixel is invalid, and the valid depth value will be used in subsequent applications such as 3D reconstruction.

请参阅图5,本申请除了通过识别目标场景的类型,自适应设置置信度阈值之外,还可以通过时间滤波的方式进一步确定深度像素是否有效,检测方法还包括:Referring to FIG. 5 , in addition to adaptively setting the confidence threshold by identifying the type of the target scene, the present application can further determine whether the depth pixel is valid by means of temporal filtering, and the detection method further includes:

步骤015:根据当前深度图像的前N帧的深度像素的深度值、置信度值和有效次数,生成历史帧,N为正整数,有效次数根据当前深度图像的前N帧中对应位置的深度像素的置信度值和置信度阈值的比较结果确定。Step 015: According to the depth value, confidence value and valid times of the depth pixels of the first N frames of the current depth image, generate a historical frame, N is a positive integer, and the valid times are based on the depth pixels of the corresponding positions in the first N frames of the current depth image. The confidence value and the confidence threshold are compared to determine the result.

具体地,通过当前深度图像的前N帧深度图像,融合生成一个历史帧,如将前N帧(如前4帧、前5帧等)深度图像中,相同位置的深度像素的深度值的均值或加权平均值等作为历史帧中相同位置的像素的深度值;和/或将前N帧深度图像中,相同位置的深度像素对应的置信度值的均值或加权平均值等作为历史帧中相同位置的像素的置信度值;和/或将前N帧深度图像中,相同位置的深度像素中置信度值大于置信度阈值的深度像素的个数统计出来,以作为历史帧中相同位置的像素的有效次数。如此,通过当前深度图像的前N帧深度图像,生成历史帧,以用于后续根据历史帧来判断当前帧的深度像素是否有效。Specifically, a historical frame is generated by fusing the previous N frames of depth images of the current depth image. or the weighted average, etc., as the depth value of the pixels at the same position in the historical frame; and/or the mean or weighted average of the confidence values corresponding to the depth pixels at the same position in the depth images of the previous N frames as the same in the historical frame. The confidence value of the pixel at the position; and/or count the number of depth pixels whose confidence value is greater than the confidence threshold in the depth images at the same position in the depth images of the previous N frames, as the pixel at the same position in the historical frame. Valid times. In this way, a history frame is generated through the first N frames of depth images of the current depth image, which is used to subsequently determine whether the depth pixels of the current frame are valid according to the history frames.

可以理解,对于散斑TOF相机而言,由于其成像位置是离散的,深度像素的成像位置在某个区域内变化,因此,可先确定每个深度像素对应的像素区域,然后按照每个像素区域进行历史帧的维护,如按位置对应(如位置相同)的像素区域进行融合,如将前N帧中位置相同的像素区域中的深度像素的深度值的均值或加权平均值作为该像素区域的深度值,将前N帧中位置相同的像素区域中的深度像素的置信度值的均值或加权平均值作为该像素区域的置信度值。从而根据前N帧的像素区域的深度值、置信度值和有效次数,生成历史帧。It can be understood that for the speckle TOF camera, since its imaging position is discrete, the imaging position of the depth pixel changes in a certain area. Therefore, the pixel area corresponding to each depth pixel can be determined first, and then the pixel area corresponding to each depth pixel can be determined first. For example, the pixel area corresponding to the position (such as the same position) is fused. For example, the average or weighted average of the depth values of the depth pixels in the pixel area with the same position in the previous N frames is used as the pixel area. The average or weighted average of the confidence values of the depth pixels in the pixel region with the same position in the previous N frames is taken as the confidence value of the pixel region. Thus, a historical frame is generated according to the depth value, confidence value and valid times of the pixel area of the previous N frames.

可选地,为了节省历史帧维护的计算量,可将深度图像划分为多个子区域,按位置对应的子区域进行融合,如先将子区域中的深度像素的深度值的均值或加权平均值作为该子区域的深度值,将子区域的中的深度像素的置信度值的均值或加权平均值作为该子区域的置信度值,然后根据前N帧的子区域的深度值、置信度值和有效次数,生成历史帧。Optionally, in order to save the calculation amount of historical frame maintenance, the depth image can be divided into multiple sub-regions, and the sub-regions corresponding to the positions can be fused. For example, the average or weighted average of the depth values of the depth pixels in the sub-region As the depth value of the sub-area, the mean or weighted average of the confidence values of the depth pixels in the sub-area is taken as the confidence value of the sub-area, and then according to the depth value and confidence value of the sub-area in the previous N frames and the number of valid times to generate a history frame.

请再次参阅图5,在确定历史帧后,可根据历史帧、置信度值和置信度阈值检测深度像素是否有效,步骤014:根据置信度值和置信度阈值检测深度像素是否有效,具体包括:Please refer to FIG. 5 again, after determining the historical frame, it is possible to detect whether the depth pixel is valid according to the historical frame, the confidence value and the confidence threshold. Step 014: Detect whether the depth pixel is valid according to the confidence value and the confidence threshold, specifically including:

步骤0141:在当前深度像素在历史帧中位置对应的目标深度像素的有效次数大于预设次数、且当前深度像素的置信度值大于置信度阈值的情况下,确定当前深度像素有效;或者Step 0141: In the case where the valid times of the target depth pixel corresponding to the position of the current depth pixel in the historical frame is greater than the preset number of times, and the confidence value of the current depth pixel is greater than the confidence threshold, determine that the current depth pixel is valid; or

步骤0142:在当前深度像素在历史帧中位置对应的目标深度像素的有效次数大于预设次数、且当前深度像素的置信度值小于置信度阈值的情况下,使用目标深度像素的目标深度值和当前深度像素的当前深度值重新加权生成当前深度值,当前深度值的权值和目标深度值的权值根据当前深度像素和目标深度像素的深度值差值和置信度值差值中至少一个确定;或者Step 0142: When the valid times of the target depth pixel corresponding to the position of the current depth pixel in the historical frame is greater than the preset number of times, and the confidence value of the current depth pixel is less than the confidence threshold, use the target depth value of the target depth pixel and The current depth value of the current depth pixel is re-weighted to generate the current depth value. The weight of the current depth value and the weight of the target depth value are determined according to at least one of the difference between the depth value and the confidence value difference between the current depth pixel and the target depth pixel. ;or

步骤0143:在当前深度像素在历史帧中位置对应的目标深度像素的有效次数小于预设次数的情况下,确定当前深度像素无效;或者Step 0143: In the case that the valid times of the target depth pixel corresponding to the position of the current depth pixel in the historical frame is less than the preset number of times, determine that the current depth pixel is invalid; or

步骤0144:在当前深度像素和置信度阈值的差值小于第三预设差值阈值、且当前深度像素在历史帧中位置对应的目标深度像素的有效次数小于预设次数的情况下,确定当前深度像素无效。Step 0144: In the case that the difference between the current depth pixel and the confidence threshold is less than the third preset difference threshold, and the valid times of the target depth pixel corresponding to the position of the current depth pixel in the historical frame is less than the preset number of times, determine the current depth. Invalid depth pixel.

具体地,根据策略的不同,可选择不同的方式确定深度像素是否有效。Specifically, according to different strategies, different ways can be selected to determine whether the depth pixels are valid.

策略一:完整性优先,只增不减。Strategy 1: Integrity first, only increase but not decrease.

在当前深度像素在历史帧中位置对应(如位置相同)的目标深度像素的有效次数大于预设次数(如N/2,N与上述生成历史帧的N帧的数值相同)时,则当前深度像素应当保留或增加。保留的含义是,若当前深度像素的置信度值大于或等于置信度阈值,则确定当前深度像素有效;增加的含义是,若当前深度像素的置信度值小于置信度阈值,则当前深度像素的深度值可能是不可靠或者不准确的,即采用目标深度像素的深度值替换当前深度像素的深度值。或者,根据与当前深度像素邻近的深度像素的深度值的平均值或加权平均值来替换当前深度像素的深度值,其中,与当前深度像素邻近的深度像素的深度值的权重可根据该深度像素和当前深度像素的像素距离确定,比如(1,1)点和(1,2)点,像素距离就是1,像素距离越大,权重则越小。When the valid times of the target depth pixel corresponding to the position of the current depth pixel in the historical frame (such as the same position) is greater than the preset times (such as N/2, N is the same as the value of the N frame of the above-mentioned historical frame generation), then the current depth Pixels should be preserved or increased. The meaning of retention is that if the confidence value of the current depth pixel is greater than or equal to the confidence threshold, it is determined that the current depth pixel is valid; the meaning of increase is that if the confidence value of the current depth pixel is less than the confidence threshold, then the current depth pixel is valid. The depth value may be unreliable or inaccurate, that is, the depth value of the current depth pixel is replaced with the depth value of the target depth pixel. Alternatively, the depth value of the current depth pixel is replaced according to the average or weighted average value of the depth values of the depth pixels adjacent to the current depth pixel, wherein the weight of the depth values of the depth pixels adjacent to the current depth pixel may be based on the depth pixel. The pixel distance from the current depth pixel is determined, such as (1,1) point and (1,2) point, the pixel distance is 1, and the larger the pixel distance, the smaller the weight.

也即是说,在当前深度像素在历史帧中位置对应的目标深度像素的有效次数大于预设次数的情况下,不论当前深度像素的置信度值是否大于置信度阈值,深度像素均存在对应的深度值,即深度像素均有效。That is to say, in the case that the valid times of the target depth pixel corresponding to the position of the current depth pixel in the historical frame is greater than the preset number of times, no matter whether the confidence value of the current depth pixel is greater than the confidence threshold, there is a corresponding depth pixel. Depth values, i.e. depth pixels, are valid.

其中,这里的N/2仅是举例,实际使用的时候可以根据需要设置N/3、3N/4等,可以理解,预设次数越大,该点保留的可能性越低,所以该值要根据实际所需完整性与准确性折中之后进行取值。Among them, N/2 here is just an example. In actual use, N/3, 3N/4, etc. can be set as needed. It is understandable that the larger the preset number of times, the lower the possibility of the point being retained, so the value should be The value is taken after a compromise between the actual required integrity and accuracy.

可选地,在当前深度像素在历史帧中位置对应(如位置相同)的目标深度像素的有效次数大于预设次数且当前深度像素的置信度值小于置信度阈值的情况下,除了可以直接用历史帧相同位置的深度值代替当前深度像素的深度值外,也可以当前深度像素的当前深度值与目标深度像素的目标深度值做加权平均后的值作为当前深度值,其中,当前深度值的权值和目标深度值的权值根据当前深度像素和目标深度像素的深度值差值和置信度值差值中至少一个确定。例如,深度值差值越大,说明环境发生改变的可能性较大,则增加当前深度值的权值并降低目标深度值的权值,再例如,置信度值差值越大,说明环境发生改变的可能性也较大,则增加当前深度值的权值并降低目标深度值的权值。如此,可在保证完整性的前提下,尽可能的提升当前深度像素的深度值的准确性,且静止场景下的准确性更高。Optionally, when the valid times of the target depth pixel corresponding to the position of the current depth pixel in the historical frame (such as the same position) is greater than the preset number of times and the confidence value of the current depth pixel is less than the confidence threshold, in addition to directly using In addition to replacing the depth value of the current depth pixel with the depth value at the same position of the historical frame, the current depth value of the current depth value of the current depth pixel and the target depth value of the target depth pixel can also be weighted and averaged as the current depth value. The weight value and the weight value of the target depth value are determined according to at least one of a depth value difference and a confidence value difference between the current depth pixel and the target depth pixel. For example, if the difference between the depth values is larger, it indicates that the environment is more likely to change, so the weight of the current depth value is increased and the weight of the target depth value is decreased. If the possibility of change is also high, the weight of the current depth value is increased and the weight of the target depth value is decreased. In this way, on the premise of ensuring completeness, the accuracy of the depth value of the current depth pixel can be improved as much as possible, and the accuracy in a static scene is higher.

可以理解的是,由于历史帧的深度值是之前时刻的深度距离,且由于TOF相机位置或视角变换之后,当前深度像素对应的三维真实距离可能与目标深度值或前述加权平均后的值不同,使用目标深度值替换当前深度值,或者使用当前深度像素的当前深度值与目标深度像素的目标深度值做加权平均后的值替换当前深度值时,容易造成拖影。It can be understood that since the depth value of the historical frame is the depth distance at the previous moment, and due to the TOF camera position or perspective transformation, the three-dimensional real distance corresponding to the current depth pixel may be different from the target depth value or the aforementioned weighted average value. When replacing the current depth value with the target depth value, or replacing the current depth value with the weighted average value of the current depth value of the current depth pixel and the target depth value of the target depth pixel, it is easy to cause smear.

可选地,在当前深度像素在历史帧中位置对应(如位置相同)的目标深度像素的有效次数大于预设次数情况下,也可直接保留当前深度值,此时历史帧的深度值不发挥作用,仅有效次数发挥作用,可以避免拖影问题。或者,在使用当前深度像素的当前深度值与目标深度像素的目标深度值做加权平均后的值替换当前深度值时,适当增加当前深度值的权值可降低拖影现象。Optionally, in the case that the valid times of the target depth pixel corresponding to the position of the current depth pixel in the historical frame (such as the same position) is greater than the preset number of times, the current depth value can also be directly retained, and the depth value of the historical frame does not play a role at this time. function, only the effective times play a role, which can avoid the smear problem. Alternatively, when the current depth value is replaced by a weighted average value of the current depth value of the current depth pixel and the target depth value of the target depth pixel, appropriately increasing the weight of the current depth value can reduce the smear phenomenon.

策略二:准确性优先,只减不加。Strategy 2: Accuracy is given priority, only subtraction but not addition.

在当前深度像素在历史帧中位置对应的目标深度像素的有效次数小于或等于预设次数(如N/2)的情况下,则不保留该点,确定当前深度像素无效。或者,当前深度像素在置信度阈值M的预设范围内波动时,如预设范围为[M-1,M+1],确定当前深度像素可能发生闪烁的情况下,再判断目标深度像素的有效次数是否小于或等于预设次数(如N/2),若是,则确定当前深度像素无效,从而提升判断准确性。In the case that the valid times of the target depth pixel corresponding to the position of the current depth pixel in the historical frame is less than or equal to a preset number of times (eg, N/2), the point is not retained, and the current depth pixel is determined to be invalid. Or, when the current depth pixel fluctuates within the preset range of the confidence threshold M, such as the preset range is [M-1, M+1], it is determined that the current depth pixel may flicker, and then determine the target depth pixel. Whether the valid number of times is less than or equal to the preset number of times (such as N/2), if so, it is determined that the current depth pixel is invalid, thereby improving the accuracy of judgment.

可以理解的是,策略二没有用到历史帧的深度值,只用到有效次数决定当前深度像素的当前深度值是否保留,因此不存在拖影问题。It is understandable that the second strategy does not use the depth value of the historical frame, and only uses the valid times to determine whether the current depth value of the current depth pixel is retained, so there is no smear problem.

策略三:兼顾完整性与准确性,有加有减。Strategy 3: Taking into account the integrity and accuracy, there are additions and subtractions.

在当前深度像素在历史帧中位置对应的目标深度像素的有效次数大于预设次数、且当前深度像素的置信度值小于置信度阈值的情况下,已经确定了当前深度像素有效,请参阅策略一。此时为了提高当前深度像素的深度值的准确性,可综合考虑当前深度像素和目标深度像素的深度值差值和置信度值差值,若深度值差值和置信度值差值中至少一个较大,以深度值差值和置信度值差值均较大(如深度值差值大于当前深度像素的深度值的20%、30%等,置信度值差值当前深度像素的置信度值的20%、30%等)为例,则可确定当前深度图像的场景可能发生变化,此时采用历史帧的目标深度像素的深度值可能会影响准确性,则可直接采用当前深度值,反之(即深度值差值和置信度值差值均较小),则采用目标深度像素的深度值来替换当前深度像素的深度值。When the number of valid times of the target depth pixel corresponding to the position of the current depth pixel in the historical frame is greater than the preset number of times, and the confidence value of the current depth pixel is smaller than the confidence threshold, it has been determined that the current depth pixel is valid, please refer to Strategy 1 . At this time, in order to improve the accuracy of the depth value of the current depth pixel, the depth value difference and confidence value difference between the current depth pixel and the target depth pixel can be comprehensively considered. If at least one of the depth value difference and the confidence value difference If the difference between the depth value and the confidence value is larger (for example, the difference between the depth value is greater than 20%, 30% of the depth value of the current depth pixel, etc., the difference value of the confidence value is the confidence value of the current depth pixel. 20%, 30%, etc.) as an example, it can be determined that the scene of the current depth image may change. At this time, the depth value of the target depth pixel of the historical frame may affect the accuracy, and the current depth value can be directly used, and vice versa. (that is, both the depth value difference and the confidence value difference are small), the depth value of the target depth pixel is used to replace the depth value of the current depth pixel.

在当前深度像素在历史帧中位置对应的目标深度像素的有效次数小于预设次数(如N/2)、且当前深度像素的置信度值小于置信度阈值的情况下,即表示当前深度值本身的准确性较低的情况下,历史多帧中对应的深度值的准确性也较低,因此当前深度值出现闪烁的几率较大,从而确定当前深度像素无效,防止当前深度像素闪烁。本策略同时考虑了完整性与准确率。In the case that the valid times of the target depth pixel corresponding to the position of the current depth pixel in the historical frame is less than a preset number of times (such as N/2), and the confidence value of the current depth pixel is less than the confidence threshold, it means that the current depth value itself When the accuracy of the current depth value is low, the accuracy of the corresponding depth value in the historical multiple frames is also low, so the current depth value has a high probability of flickering, so that the current depth pixel is determined to be invalid and the current depth pixel is prevented from flickering. This strategy considers both completeness and accuracy.

在根据历史帧对当前深度图像的深度像素进行有效性判断后,检测方法还包括:After judging the validity of the depth pixels of the current depth image according to the historical frames, the detection method further includes:

请参阅图6,步骤016:可根据当前深度图像的深度值和置信度值,来更新历史帧。Please refer to FIG. 6, step 016: the historical frame can be updated according to the depth value and confidence value of the current depth image.

具体地,在更新历史帧时,若当前深度像素和目标深度像素的深度值差值和置信度值差值中任意一个大于第三预设差值阈值的情况下,可在生成当前深度图像的下一帧的历史帧时,增加当前深度图像的权值,从而使得历史帧中的像素点的深度值、置信度值和有效次数受到当前深度图的影响增加,从而提高历史帧的准确性。Specifically, when updating the historical frame, if any one of the difference between the depth value and the confidence value difference between the current depth pixel and the target depth pixel is greater than the third preset difference threshold, the current depth image can be generated in the In the history frame of the next frame, the weight of the current depth image is increased, so that the depth value, confidence value and valid times of the pixels in the history frame are increased by the influence of the current depth map, thereby improving the accuracy of the history frame.

更具体地,在当前深度图像的深度像素的第一深度值和历史帧中对应位置的深度像素(即目标深度像素)的第二深度值的差值大于第一预设差值阈值(如第一预设差值阈值为第一深度值的20%、30%等)的情况下,根据第一深度值和第二深度值加权生成第三深度值,以替换第二深度值,第一深度值的权重大于第二深度值的权重,第一深度值的权重和第二深度值的权重根据第一深度值的权重和第二深度值的差值确定,该差值越大,则第一深度值的权重越大,而第二深度值的权重越小;和/或More specifically, the difference between the first depth value of the depth pixel of the current depth image and the second depth value of the depth pixel (that is, the target depth pixel) at the corresponding position in the historical frame is greater than the first preset difference threshold (such as the first When a preset difference threshold is 20%, 30%, etc. of the first depth value, a third depth value is weighted according to the first depth value and the second depth value to replace the second depth value, the first depth value The weight of the value is greater than the weight of the second depth value. The weight of the first depth value and the weight of the second depth value are determined according to the difference between the weight of the first depth value and the second depth value. The greater the weight of the depth value and the lesser the weight of the second depth value; and/or

在当前深度图像的深度像素的第一置信度值和历史帧中对应位置的深度像素(即目标深度像素)的第二置信度值的差值大于第二预设差值阈值(如第二预设差值阈值为第二深度值的20%、30%等)的情况下,根据第一置信度值和第二置信度值加权生成第三置信度值,以替换第二置信度值,第一置信度值的权重大于第二置信度值的权重,第一置信度值的权重和第二置信度值的权重根据第一置信度值的权重和第二置信度值的差值确定,该差值越大,则第一置信度值的权重越大,而第二置信度值的权重越小。The difference between the first confidence value of the depth pixel of the current depth image and the second confidence value of the depth pixel at the corresponding position in the historical frame (ie the target depth pixel) is greater than the second preset difference threshold (such as the second preset difference threshold). If the difference threshold is set to be 20%, 30% of the second depth value, etc.), a third confidence value is weighted according to the first confidence value and the second confidence value to replace the second confidence value. The weight of a confidence value is greater than the weight of the second confidence value, and the weight of the first confidence value and the weight of the second confidence value are determined according to the difference between the weight of the first confidence value and the second confidence value. The larger the difference, the larger the weight of the first confidence value, and the smaller the weight of the second confidence value.

可以理解,前述提到的第一预设差值阈值、第二预设差值阈值及第三预设差值阈值均为经验值,可根据实际情况确定。It can be understood that the first preset difference threshold, the second preset difference threshold and the third preset difference threshold mentioned above are all empirical values and can be determined according to actual conditions.

在自适应设置置信度阈值和时间滤波的方式进一步确定深度像素是否有效时,还可结合空间滤波的方式进一步提升当前深度图像的深度值和置信度值的准确性,并确定当前深度图像中的深度像素是否有效,针对当前深度图像的深度值和置信度值的准确性,检测方法还包括:When adaptively setting the confidence threshold and temporal filtering to further determine whether the depth pixel is valid, the spatial filtering method can be combined to further improve the accuracy of the depth value and confidence value of the current depth image, and determine the depth value in the current depth image. Whether the depth pixel is valid, for the accuracy of the depth value and confidence value of the current depth image, the detection method also includes:

请参阅图7,步骤017:根据当前深度像素周围预设尺寸范围内的深度像素的深度值和置信度值,对当前深度像素进行滤波,以更新当前深度像素的深度值和置信度值。Referring to FIG. 7, step 017: Filter the current depth pixel according to the depth value and confidence value of the depth pixel within a preset size range around the current depth pixel to update the depth value and confidence value of the current depth pixel.

具体地,在更新深度值时,根据当前深度像素周围预设尺寸范围,如以当前深度像素为中心的3*3像素的范围内,一共9个深度像素的深度值和置信度值,来对当前深度像素的深度值进行滤波,如确定9个深度像素的深度值的均值或加权平均值为当前深度像素的当前深度值,或者,可仅根据9个深度像素中,置信度值大于置信度阈值的深度像素的深度值的均值或加权平均值为当前深度像素的当前深度值,从而进一步提高当前深度值的准确性。Specifically, when updating the depth value, according to the preset size range around the current depth pixel, such as the range of 3*3 pixels centered on the current depth pixel, the depth values and confidence values of a total of 9 depth pixels are used to determine The depth value of the current depth pixel is filtered, for example, the average or weighted average of the depth values of 9 depth pixels is determined as the current depth value of the current depth pixel, or, it can only be based on the confidence value of the 9 depth pixels. The average or weighted average of the depth values of the thresholded depth pixels is the current depth value of the current depth pixel, thereby further improving the accuracy of the current depth value.

同样地,在更新置信度值时,根据当前深度像素周围预设尺寸范围,如以当前深度像素为中心的3*3像素的范围内,一共9个深度像素的深度值和置信度值,来对当前深度像素的置信度值进行滤波,如确定9个深度像素的置信度值的均值或加权平均值为当前深度像素的当前置信度值,或者,可仅根据9个深度像素中,置信度值大于置信度阈值的深度像素的置信度值的均值或加权平均值为当前深度像素的当前置信度值,从而进一步提高当前置信度值的准确性,进而提高根据置信度值的直方图确定的目标场景的类型的准确性。Similarly, when updating the confidence value, according to the preset size range around the current depth pixel, such as the range of 3*3 pixels with the current depth pixel as the center, the depth value and confidence value of a total of 9 depth pixels are obtained. Filter the confidence value of the current depth pixel, such as determining the mean or weighted average of the confidence values of the 9 depth pixels as the current confidence value of the current depth pixel, or, it can only be based on the confidence value of the 9 depth pixels. The mean or weighted average of the confidence values of the depth pixels whose value is greater than the confidence threshold is the current confidence value of the current depth pixel, thereby further improving the accuracy of the current confidence value, and further improving the accuracy determined according to the histogram of the confidence value. The accuracy of the type of target scene.

可选地,当前深度像素可以是置信度值小于置信度阈值的深度像素,在减少计算量的同时提高准确性。当前深度像素可以是任一深度像素,以提高每个深度像素的准确性。Optionally, the current depth pixel may be a depth pixel whose confidence value is less than the confidence threshold, which reduces the amount of calculation and improves the accuracy. The current depth pixel can be any depth pixel to improve the accuracy of each depth pixel.

请参阅图7和图8,在保证完整性的前提下(即尽可能多地保证深度像素都确定为有效),步骤014还包括:Please refer to FIG. 7 and FIG. 8, under the premise of ensuring completeness (that is, ensuring that as many depth pixels as possible are determined to be valid), step 014 further includes:

步骤0171:统计当前深度像素周围预设尺寸范围内的深度像素的置信度与当前深度像素的置信度的差值小于预定差值的目标深度像素的数量;Step 0171: Count the number of target depth pixels where the difference between the confidence level of the depth pixel within the preset size range around the current depth pixel and the confidence level of the current depth pixel is less than a predetermined difference;

步骤0172:若数量大于第一预设数量,则根据当前深度像素的深度值和目标深度像素的深度值确定滤波深度值、并根据当前深度像素的置信度值和目标深度像素的置信度值确定滤波置信度值;Step 0172: If the number is greater than the first preset number, then determine the filter depth value according to the depth value of the current depth pixel and the depth value of the target depth pixel, and determine according to the confidence value of the current depth pixel and the confidence value of the target depth pixel. filter confidence value;

步骤0173,将当前深度像素周围预设尺寸范围内的所有深度像素的深度值均替换为滤波深度值、并将当前深度像素周围预设尺寸范围内的所有深度像素的置信度值均替换为滤波置信度值。Step 0173, replace the depth values of all the depth pixels in the preset size range around the current depth pixel with the filtered depth value, and replace the confidence values of all the depth pixels in the preset size range around the current depth pixel with the filter depth value. confidence value.

具体地,对于相邻的深度像素而言,若彼此之间的置信度相近,则表示这些深度像素大概率属于同一个物体或同一个局部(如眼睛的瞳孔、嘴唇等),因此,可当前深度像素周围预设尺寸范围内的深度像素的置信度与当前深度像素的置信度的差值小于预定差值(如预定差值为经验值或者为当前深度值的5%、10%等)的目标深度像素的数量,并判断该数量是否大于第一预设数量(如预设尺寸范围为3*3像素大小),第一预设数量可以是5、6、7、8等,若是,则可确定当前深度像素的预设尺寸范围内的深度像素属于同一个物体或同一个局部,预设尺寸范围内的深度像素的深度值及置信度值均应基本相同,此时可计算当前深度像素周围预设尺寸范围内的所有深度像素的深度值的均值或中值,以生成滤波深度值,并计算当前深度像素周围预设尺寸范围内的所有深度像素的置信度值的均值或中值,以生成滤波置信度值,从而使用滤波深度值替换当前深度像素的深度值或预设尺寸范围内的所有深度像素的深度值,使用滤波置信度值替换当前深度像素的置信度值或预设尺寸范围内的所有深度像素的置信度值,在保证每个深度像素的完整性的前提下,提高深度值和置信度值的准确性。Specifically, for adjacent depth pixels, if the confidence levels are similar to each other, it means that these depth pixels have a high probability of belonging to the same object or the same part (such as the pupil of the eye, lips, etc.), so the current The difference between the confidence level of the depth pixel within the preset size range around the depth pixel and the confidence level of the current depth pixel is less than a predetermined difference value (for example, the predetermined difference value is an empirical value or is 5%, 10% of the current depth value, etc.) The number of target depth pixels, and determine whether the number is greater than the first preset number (for example, the preset size range is 3*3 pixel size), the first preset number can be 5, 6, 7, 8, etc., if so, then It can be determined that the depth pixels within the preset size range of the current depth pixel belong to the same object or the same part, the depth value and confidence value of the depth pixels within the preset size range should be basically the same, and the current depth pixel can be calculated at this time. the mean or median value of the depth values of all the depth pixels within the surrounding preset size range to generate a filtered depth value, and calculate the mean or median value of the confidence values of all the depth pixels within the preset size range around the current depth pixel, To generate a filtering confidence value, replace the depth value of the current depth pixel or the depth value of all depth pixels within the preset size range with the filtering depth value, and replace the confidence value of the current depth pixel or the preset size with the filtering confidence value The confidence value of all depth pixels within the range, on the premise of ensuring the integrity of each depth pixel, improves the accuracy of the depth value and confidence value.

可选地,可计算当前深度像素和当前深度像素周围预设尺寸范围内的所有目标深度像素的深度值的均值及置信度值的均值,以分别生成滤波深度值和滤波置信度值,从而利用准确性较高的目标深度像素的深度值,进一步提高深度值和置信度值的准确性。Optionally, the mean value of the depth value and the mean value of the confidence value of the current depth pixel and all target depth pixels within the preset size range around the current depth pixel can be calculated to generate the filtered depth value and the filtered confidence value respectively, so as to utilize The depth value of the target depth pixel with higher accuracy further improves the accuracy of the depth value and confidence value.

可选地,为了更为准确地确定预设尺寸范围,可以借助可见光图像来识别属于同一个物体或同一个局部的像素区域,以作为预设尺寸范围。Optionally, in order to more accurately determine the preset size range, a visible light image may be used to identify pixel regions belonging to the same object or the same local area as the preset size range.

首先,可确定可见光图像中与当前深度像素位置对应(即位置相同)的目标可见光像素。First, the target visible light pixel in the visible light image corresponding to (ie, the same position) as the current depth pixel position may be determined.

通常,TOF相机会与可见光相机组合在一起使用,可以看到,现在很多手机后置都不止有一个摄像头。不过,相比可见光相机,TOF作为主动光距离探测设备,功耗通常较高,且分辨率一般远低于目前市面上的可见光相机。在出厂时,通常会将可见光相机与TOF相机做标定,这样可以获取两个相机坐标系之间变换关系,也可以根据空间中某点在一个相机中成像的位置计算出该点在另一个相机坐标系中的位置。如此,可根据两个相机坐标系之间变换关系确定与当前深度像素位置对应的目标可见光像素。Usually, TOF cameras are used in combination with visible light cameras. It can be seen that many mobile phones now have more than one camera on the rear. However, compared with visible light cameras, TOF, as an active light distance detection device, usually has higher power consumption, and the resolution is generally much lower than that of visible light cameras currently on the market. When leaving the factory, the visible light camera and the TOF camera are usually calibrated, so that the transformation relationship between the two camera coordinate systems can be obtained, and the image position of a point in space in one camera can be calculated. position in the coordinate system. In this way, the target visible light pixel corresponding to the current depth pixel position can be determined according to the transformation relationship between the two camera coordinate systems.

然后,判断目标可见光像素周围预定尺寸范围内的可见光像素中,像素值与目标可见光像素的像素值的差值小于第四预设差值阈值的相似可见光像素。Then, it is determined that among the visible light pixels within a predetermined size range around the target visible light pixel, the difference between the pixel value and the pixel value of the target visible light pixel is smaller than a similar visible light pixel with a fourth preset difference threshold.

最后,根据相似可见光像素重新确定预设尺寸范围,如确定目标可见光像素和相似可见光像素覆盖的像素区域为目标区域,然后根据目标区域重新确定预设尺寸范围,如将当前深度图像中与目标区域位置相同的像素区域设置为预设尺寸范围,如此,可提高预设尺寸范围的准确性,进一步提高更新后的深度像素的深度值的准确性。Finally, re-determine the preset size range according to the similar visible light pixels, for example, determine the target visible light pixel and the pixel area covered by the similar visible light pixels as the target area, and then re-determine the preset size range according to the target area, such as comparing the current depth image with the target area The pixel area with the same position is set as the preset size range, so that the accuracy of the preset size range can be improved, and the accuracy of the depth value of the updated depth pixel can be further improved.

可选地,在完成所有深度像素的深度值的空间滤波后,可根据可见光图像识别目标物体应属于同一深度范围的多个局部(对于人脸而言,如眼睛区域、嘴唇区域、额头区域等应属于同一深度范围),从而确定多个深度范围,然后获取每个局部的深度像素的深度值及对应的深度范围,将该局部中深度值超过深度范围的深度值进行修正,以使得每个局部的深度像素的深度值均位于对应的深度范围内,进一步提高深度值的准确性。Optionally, after completing the spatial filtering of the depth values of all depth pixels, it is possible to identify multiple parts of the target object that should belong to the same depth range according to the visible light image (for a human face, such as the eye area, the lip area, the forehead area, etc. should belong to the same depth range), so as to determine multiple depth ranges, and then obtain the depth value of each local depth pixel and the corresponding depth range, and correct the depth value of the local depth value exceeding the depth range, so that each local depth value exceeds the depth range. The depth values of the local depth pixels are all located in the corresponding depth range, which further improves the accuracy of the depth values.

可选地,为了进一步保证深度像素的完整性,在深度像素不存在深度值(在生成深度图像时,数据不足无法计算出深度值)的情况下,依旧可以通过步骤017的方式确定缺失深度值的深度像素的深度值。Optionally, in order to further ensure the integrity of the depth pixel, in the case that the depth pixel does not have a depth value (when the depth image is generated, the depth value cannot be calculated due to insufficient data), the missing depth value can still be determined by means of step 017. The depth value of the depth pixel.

请参阅图9,在保证深度像素的准确性的前提下,可根据空间滤波的方式确定深度像素是否有效,步骤014:根据置信度值和置信度阈值检测深度像素是否有效,还包括:Please refer to FIG. 9 , on the premise of ensuring the accuracy of the depth pixels, it is possible to determine whether the depth pixels are valid according to the method of spatial filtering. Step 014: Detect whether the depth pixels are valid according to the confidence value and the confidence threshold, and also include:

步骤0145:统计当前深度像素周围预设尺寸范围内的所有深度像素中,置信度值大于置信度阈值的深度像素的数量;Step 0145: Count the number of depth pixels whose confidence value is greater than the confidence threshold among all the depth pixels within the preset size range around the current depth pixel;

步骤0146:在数量小于第二预设数量的情况下,确定当前深度像素无效;Step 0146: when the number is less than the second preset number, determine that the current depth pixel is invalid;

步骤0147:在数量大于第三预设数量的情况下,确定当前深度像素有效。Step 0147: In the case that the number is greater than the third preset number, determine that the current depth pixel is valid.

具体地,可根据当前深度像素围预设尺寸范围内的所有深度像素的置信度值来确定当前深度像素是否有效。例如,当前深度像素围预设尺寸范围内的所有深度像素中,置信度大于置信度阈值的深度像素的数量小于第二预设数量(如第二预设数量为预设尺寸范围内的所有深度像素的数量的1/3),则表示预设尺寸范围内的深度像素的准确性均较低,当前深度像素大概率存在闪烁现象,故将当前深度像素确定为无效像素。再例如,当前深度像素围预设尺寸范围内的所有深度像素中,置信度大于置信度阈值的深度像素的数量大于第三预设数量(如第三预设数量为预设尺寸范围内的所有深度像素的数量的2/3),则表示预设尺寸范围内的深度像素的准确性均较高,当前深度像素的深度值准确性也较高,故将当前深度像素确定为有效像素。Specifically, whether the current depth pixel is valid may be determined according to confidence values of all depth pixels within a preset size range of the current depth pixel. For example, among all the depth pixels within the preset size range of the current depth pixel, the number of depth pixels whose confidence is greater than the confidence threshold is less than the second preset number (for example, the second preset number is all depth pixels within the preset size range) 1/3 of the number of pixels), it means that the accuracy of the depth pixels within the preset size range is low, and the current depth pixels have a high probability of flickering, so the current depth pixels are determined as invalid pixels. For another example, among all the depth pixels within the preset size range of the current depth pixel, the number of depth pixels whose confidence is greater than the confidence threshold is greater than the third preset number (for example, the third preset number is all the depth pixels within the preset size range). 2/3 of the number of depth pixels), it means that the accuracy of the depth pixels within the preset size range is high, and the accuracy of the depth value of the current depth pixel is also high, so the current depth pixel is determined as an effective pixel.

需要说明的是,空间域滤波可以与时间域滤波相结合,共同判断深度像素是否有效。例如,在当前深度像素在历史帧中位置对应的目标深度像素的有效次数小于预设次数,且预设尺寸范围内的所有深度像素中,置信度值大于置信度阈值的深度像素的数量小于第二预设数量的情况下,确定当前深度像素无效,从而更为准确地确定当前深度像素是否无效。再例如,在当前深度像素在历史帧中位置对应的目标深度像素的有效次数大于预设次数,或预设尺寸范围内的所有深度像素中,置信度值大于置信度阈值的深度像素的数量小于第二预设数量的情况下,确定当前深度像素有效,且使用时间滤波或空间滤波的方式修正当前深度像素的深度值。即上述所有的技术方案,可以采用任意结合形成本申请的可选实施例,在此不再一一赘述。It should be noted that the spatial domain filtering can be combined with the temporal domain filtering to jointly determine whether the depth pixels are valid. For example, when the valid times of the target depth pixel corresponding to the position of the current depth pixel in the historical frame is less than the preset number of times, and among all the depth pixels within the preset size range, the number of depth pixels whose confidence value is greater than the confidence threshold is less than the number of In the case of two preset numbers, it is determined that the current depth pixel is invalid, so as to more accurately determine whether the current depth pixel is invalid. For another example, the number of valid times of the target depth pixel corresponding to the position of the current depth pixel in the historical frame is greater than the preset number of times, or among all the depth pixels within the preset size range, the number of depth pixels whose confidence value is greater than the confidence threshold is less than In the case of the second preset number, it is determined that the current depth pixel is valid, and the depth value of the current depth pixel is corrected by means of temporal filtering or spatial filtering. That is, all the above technical solutions can be combined arbitrarily to form optional embodiments of the present application, which will not be repeated here.

本申请实施方式的检测方法通过拍摄目标场景获取的深度图像的置信度值的分布规律,来确定目标场景的类型,如目标场景为大物体类型、大物体大背景类型、通用类型等,从而根据目标场景的类型,准确地确定适应当前类型的置信度阈值,并根据较为准确地置信度阈值来准确地判断深度像素是否有效,从而降低甚至消除像素点闪烁现象。且本申请结合空间滤波和时间滤波,不仅进一步提高了深度像素有效性的判断准确性,而且可以通过滤波提高深度值的准确性。The detection method of the embodiment of the present application determines the type of the target scene by the distribution law of the confidence value of the depth image obtained by shooting the target scene, for example, the target scene is a large object type, a large object background type, a general type, etc. The type of the target scene, accurately determine the confidence threshold suitable for the current type, and accurately judge whether the depth pixel is valid according to the more accurate confidence threshold, thereby reducing or even eliminating pixel flickering phenomenon. In addition, the present application combines spatial filtering and temporal filtering, which not only further improves the accuracy of judging the validity of depth pixels, but also can improve the accuracy of depth values through filtering.

为便于更好的实施本申请实施例的检测方法,本申请实施例还提供一种检测装置10。请参阅图10,图10为本申请实施例提供的检测装置10的结构示意图。其中,该检测装置10可以包括:To facilitate better implementation of the detection method of the embodiment of the present application, the embodiment of the present application further provides a detection device 10 . Please refer to FIG. 10 , which is a schematic structural diagram of a detection apparatus 10 provided by an embodiment of the present application. Wherein, the detection device 10 may include:

拍摄模块11,用于拍摄目标场景以获取深度图像,深度图像包括多个深度像素,每个深度像素的深度值均存在对应的置信度值,置信度值根据深度像素对应的光接收器接收的光量确定;The shooting module 11 is used for shooting a target scene to obtain a depth image, the depth image includes a plurality of depth pixels, and the depth value of each depth pixel has a corresponding confidence value, and the confidence value is based on the light receiver corresponding to the depth pixel. Determine the amount of light;

识别模块12,用于根据置信度值识别目标场景的类型;The identification module 12 is used for identifying the type of the target scene according to the confidence value;

识别模块12具体用于:The identification module 12 is specifically used for:

根据所有深度像素的置信度值生成置信度值直方图;Generate a confidence value histogram based on the confidence values of all depth pixels;

根据置信度值直方图识别目标场景的类型。Identify the type of target scene based on the histogram of confidence values.

识别模块12具体还用于:The identification module 12 is also specifically used for:

获取置信度值位于预设置信度值范围内的深度像素的数量占所有深度像素的数量的比例;及Obtain the ratio of the number of depth pixels whose confidence value is within the range of the preset confidence value to the number of all depth pixels; and

根据比例确定目标场景的类型。Determine the type of target scene based on the scale.

第一确定模块13,用于根据目标场景的类型确定置信度阈值;The first determination module 13 is used to determine the confidence threshold according to the type of the target scene;

第一确定模块13具体用于:The first determining module 13 is specifically used for:

根据深度像素的反射率对应的预设阈值和目标场景的类型确定每个深度像素对应的置信度阈值。The confidence threshold corresponding to each depth pixel is determined according to the preset threshold corresponding to the reflectivity of the depth pixel and the type of the target scene.

第一确定模块13具体还用于:The first determining module 13 is also specifically used for:

第二确定模块14,用于根据置信度值和置信度阈值检测深度像素是否有效。The second determination module 14 is configured to detect whether the depth pixel is valid according to the confidence value and the confidence threshold.

第二确定模块14具体用于在当前深度像素在历史帧中位置对应的目标深度像素的有效次数大于预设次数、且当前深度像素的置信度值大于置信度阈值的情况下,确定当前深度像素有效;或者The second determination module 14 is specifically configured to determine the current depth pixel when the valid times of the target depth pixel corresponding to the position of the current depth pixel in the historical frame is greater than the preset number of times and the confidence value of the current depth pixel is greater than the confidence threshold. is valid; or

在当前深度像素在历史帧中位置对应的目标深度像素的有效次数大于预设次数、且当前深度像素的置信度值小于置信度阈值的情况下,使用目标深度像素的目标深度值和当前深度像素的当前深度值重新加权生成当前深度值,当前深度值的权值和目标深度值的权值根据当前深度像素和目标深度像素的深度值差值和置信度值差值中至少一个确定;或者When the valid times of the target depth pixel corresponding to the position of the current depth pixel in the historical frame is greater than the preset times, and the confidence value of the current depth pixel is less than the confidence threshold, the target depth value of the target depth pixel and the current depth pixel are used. The current depth value is reweighted to generate the current depth value, and the weight of the current depth value and the weight of the target depth value are determined according to at least one of the depth value difference and the confidence value difference between the current depth pixel and the target depth pixel; or

在当前深度像素在历史帧中位置对应的目标深度像素的有效次数小于预设次数的情况下,确定当前深度像素无效;或者In the case that the number of valid times of the target depth pixel corresponding to the position of the current depth pixel in the historical frame is less than the preset number of times, it is determined that the current depth pixel is invalid; or

在当前深度像素在历史帧中位置对应的目标深度像素的有效次数小于预设次数、且当前深度像素的置信度值小于置信度阈值的情况下,确定当前深度像素无效。In the case that the valid times of the target depth pixel corresponding to the position of the current depth pixel in the historical frame is less than the preset number of times, and the confidence value of the current depth pixel is less than the confidence threshold, it is determined that the current depth pixel is invalid.

第二确定模块14具体还用于:The second determining module 14 is also specifically used for:

统计当前深度像素周围预设尺寸范围内的所有深度像素中,置信度值大于置信度阈值的深度像素的数量;Count the number of depth pixels whose confidence value is greater than the confidence threshold among all depth pixels within the preset size range around the current depth pixel;

在数量小于第二预设数量的情况下,确定当前深度像素无效;In the case that the number is less than the second preset number, it is determined that the current depth pixel is invalid;

在数量大于第三预设数量的情况下,确定当前深度像素有效。In the case that the number is greater than the third preset number, it is determined that the current depth pixel is valid.

检测装置10还包括修正模块15。修正模块15用于根据预设标定信息和深度像素的图像坐标,修正置信度值。The detection device 10 also includes a correction module 15 . The correction module 15 is configured to correct the confidence value according to the preset calibration information and the image coordinates of the depth pixels.

检测装置10还包括第三确定模块16和第四确定模块17。第三确定模块16用于根据每个深度像素的深度值和修正后置信度值,确定每个深度像素的反射率;第四确定模块17用于根据每个深度像素的深度值、置信度值和图像坐标,确定每个深度像素的反射率。The detection device 10 further includes a third determination module 16 and a fourth determination module 17 . The third determination module 16 is used for determining the reflectivity of each depth pixel according to the depth value and the corrected confidence value of each depth pixel; the fourth determination module 17 is used for determining the reflectivity of each depth pixel according to the depth value and confidence value of each depth pixel and image coordinates to determine the reflectivity of each depth pixel.

检测装置10还包括生成模块18。生成模块18用于根据当前深度图像的前N帧的深度像素的深度值、置信度值和有效次数,生成历史帧,N为正整数,有效次数根据当前深度图像的前N帧中对应位置的深度像素的置信度值和置信度阈值的比较结果确定。The detection device 10 also includes a generation module 18 . The generation module 18 is used to generate historical frames according to the depth value, confidence value and valid times of the depth pixels of the first N frames of the current depth image, where N is a positive integer, and the valid times are based on the corresponding position in the first N frames of the current depth image. The comparison result of the confidence value of the depth pixel and the confidence threshold is determined.

检测装置10还包括更新模块19。更新模块19用于根据当前深度图像的深度值和置信度值,更新历史帧。The detection device 10 also includes an update module 19 . The updating module 19 is configured to update the historical frame according to the depth value and confidence value of the current depth image.

更新模块19具体用于:The update module 19 is specifically used for:

在当前深度图像的深度像素的第一深度值和历史帧中对应位置的深度像素的第二深度值的差值大于第一预设差值阈值的情况下,根据第一深度值和第二深度值加权生成第三深度值,以替换第二深度值,第一深度值的权重大于第二深度值的权重,第一深度值的权重和第二深度值的权重根据第一深度值的权重和第二深度值的差值确定;和/或When the difference between the first depth value of the depth pixel of the current depth image and the second depth value of the depth pixel at the corresponding position in the historical frame is greater than the first preset difference threshold, according to the first depth value and the second depth The value is weighted to generate a third depth value to replace the second depth value, the weight of the first depth value is greater than the weight of the second depth value, the weight of the first depth value and the weight of the second depth value are based on the weight of the first depth value and A difference determination of the second depth value; and/or

在当前深度图像的深度像素的第一置信度值和历史帧中对应位置的深度像素的第二置信度值的差值大于第二预设差值阈值的情况下,根据第一置信度值和第二置信度值加权生成第三置信度值,以替换第二置信度值,第一置信度值的权重大于第二置信度值的权重,第一置信度值的权重和第二置信度值的权重根据第一置信度值的权重和第二置信度值的差值确定。In the case where the difference between the first confidence value of the depth pixel of the current depth image and the second confidence value of the depth pixel at the corresponding position in the historical frame is greater than the second preset difference threshold, according to the first confidence value and The second confidence value is weighted to generate a third confidence value to replace the second confidence value, the weight of the first confidence value is greater than the weight of the second confidence value, the weight of the first confidence value and the second confidence value The weight of is determined according to the difference between the weight of the first confidence value and the second confidence value.

检测装置10还包括滤波模块20。滤波模块20用于根据当前深度像素周围预设尺寸范围内的深度像素的深度值和置信度值,对当前深度像素进行滤波,以更新当前深度像素的深度值和置信度值。The detection device 10 further includes a filtering module 20 . The filtering module 20 is configured to filter the current depth pixel according to the depth value and confidence value of the depth pixel within a preset size range around the current depth pixel to update the depth value and confidence value of the current depth pixel.

滤波模块20具体用于:The filtering module 20 is specifically used for:

统计当前深度像素周围预设尺寸范围内的深度像素的置信度与当前深度像素的置信度的差值小于预定差值的目标深度像素的数量;Counting the number of target depth pixels where the difference between the confidence level of the depth pixel within the preset size range around the current depth pixel and the confidence level of the current depth pixel is less than the predetermined difference value;

若数量大于第一预设数量,则根据当前深度像素的深度值和目标深度像素的深度值确定滤波深度值;If the number is greater than the first preset number, the filtering depth value is determined according to the depth value of the current depth pixel and the depth value of the target depth pixel;

将当前深度像素周围预设尺寸范围内的所有深度像素的深度值均替换为滤波深度值。Replaces the depth values of all depth pixels within the preset size range around the current depth pixel with the filtered depth values.

滤波模块20具体还用于:The filtering module 20 is also specifically used for:

确定可见光图像中与当前深度像素位置对应的目标可见光像素;Determine the target visible light pixel corresponding to the current depth pixel position in the visible light image;

判断目标可见光像素周围预定尺寸范围内的可见光像素中,像素值与目标可见光像素的像素值的差值小于第四预设差值阈值的相似可见光像素;Determining that among the visible light pixels within a predetermined size range around the target visible light pixel, the difference between the pixel value and the pixel value of the target visible light pixel is smaller than a similar visible light pixel with a fourth preset difference threshold;

根据相似可见光像素重新确定预设尺寸范围。Re-determine the preset size range based on similar visible light pixels.

上述检测装置中的各个模块可全部或部分通过软件、硬件及其组合来实现。上述各个模块可以以硬件形式内嵌于或独立于计算机设备中的处理器中,也可以以软件形式存储于计算机设备中的存储器中,以便于处理器调用执行上述各个模块对应的操作。Each module in the above-mentioned detection device can be implemented in whole or in part by software, hardware and combinations thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of the above modules.

本申请实施方式的电子设备100包括处理器30。处理器30用于执行上述任意一种实施方式的检测方法,为了简洁,在此不再赘述。The electronic device 100 according to the embodiment of the present application includes the processor 30 . The processor 30 is configured to execute the detection method of any one of the foregoing embodiments, and for brevity, details are not described herein again.

其中,电子设备100可以是移动电话,智能电话,个人数字助理(personal digitalassistants,PDA),平板电脑和视频游戏设备,便携式终端(例如笔记本电脑),或较大尺寸的设备(例如台式计算机和电视),或者其他任何类型的包括TOF相机的设备。Among them, the electronic device 100 may be a mobile phone, a smart phone, a personal digital assistant (PDA), a tablet computer and a video game device, a portable terminal (such as a notebook computer), or a larger size device (such as a desktop computer and a television) ), or any other type of device that includes a TOF camera.

请参阅图7,本申请实施方式还提供了一种计算机可读存储介质300,其上存储有计算机程序310,计算机程序310被处理器30执行的情况下,实现上述任意一种实施方式的检测方法的步骤,为了简洁,在此不再赘述。Referring to FIG. 7 , an embodiment of the present application further provides a computer-readable storage medium 300 on which a computer program 310 is stored. When the computer program 310 is executed by the processor 30 , the detection of any of the foregoing embodiments is implemented. The steps of the method are not repeated here for brevity.

可以理解,计算机程序310包括计算机程序代码。计算机程序代码可以为源代码形式、对象代码形式、可执行文件或某些中间形式等。计算机可读存储介质可以包括:能够携带计算机程序代码的任何实体或装置、记录介质、U盘、移动硬盘、磁碟、光盘、计算机存储器、只读存储器(ROM,Read-Only Memory)、随机存取存储器(RAM,Random Access Memory)、以及软件分发介质等。It is understood that the computer program 310 includes computer program code. The computer program code may be in source code form, object code form, an executable file or some intermediate form, or the like. Computer-readable storage media may include: any entity or device capable of carrying computer program codes, recording media, U disk, removable hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random memory Access memory (RAM, Random Access Memory), and software distribution media, etc.

在本说明书的描述中,参考术语“一个实施方式”、“一些实施方式”、“示意性实施方式”、“示例”、“具体示例”或“一些示例”等的描述意指结合所述实施方式或示例描述的具体特征、结构、材料或者特点包含于本申请的至少一个实施方式或示例中。在本说明书中,对上述术语的示意性表述不一定指的是相同的实施方式或示例。而且,描述的具体特征、结构、材料或者特点可以在任何的一个或多个实施方式或示例中以合适的方式结合。此外,在不相互矛盾的情况下,本领域的技术人员可以将本说明书中描述的不同实施例或示例以及不同实施例或示例的特征进行结合和组合。In the description of this specification, reference to the terms "one embodiment," "some embodiments," "exemplary embodiment," "example," "specific example," or "some examples" or the like is meant to be used in conjunction with the described embodiments. A particular feature, structure, material, or characteristic described in a manner or example is included in at least one embodiment or example of the present application. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and combine the different embodiments or examples described in this specification, as well as the features of the different embodiments or examples, without conflicting each other.

流程图中或在此以其他方式描述的任何过程或方法描述可以被理解为,表示包括一个或更多个用于实现特定逻辑功能或过程的步骤的可执行指令的代码的模块、片段或部分,并且本申请的优选实施方式的范围包括另外的实现,其中可以不按所示出或讨论的顺序,包括根据所涉及的功能按基本同时的方式或按相反的顺序,来执行功能,这应被本申请的实施例所属技术领域的技术人员所理解。Any description of a process or method in the flowcharts or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing a specified logical function or step of the process , and the scope of the preferred embodiments of the present application includes alternative implementations in which the functions may be performed out of the order shown or discussed, including performing the functions substantially concurrently or in the reverse order depending upon the functions involved, which should It is understood by those skilled in the art to which the embodiments of the present application belong.

尽管上面已经示出和描述了本申请的实施方式,可以理解的是,上述实施方式是示例性的,不能理解为对本申请的限制,本领域的普通技术人员在本申请的范围内可以对上述实施方式进行变化、修改、替换和变型。Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limitations to the present application. Embodiments are subject to variations, modifications, substitutions and alterations.

Claims (17)

1.一种检测方法,其特征在于,包括:1. a detection method, is characterized in that, comprises: 拍摄目标场景以获取深度图像,所述深度图像包括多个深度像素,每个所述深度像素的深度值均存在对应的置信度值,所述置信度值根据所述深度像素对应的光接收器接收的光量确定;Shooting the target scene to obtain a depth image, the depth image includes a plurality of depth pixels, and the depth value of each of the depth pixels has a corresponding confidence value, and the confidence value is based on the light receiver corresponding to the depth pixel. The amount of light received is determined; 根据所述置信度值识别所述目标场景的类型;Identify the type of the target scene according to the confidence value; 根据所述目标场景的类型确定置信度阈值;及determining a confidence threshold based on the type of the target scene; and 根据所述置信度值和所述置信度阈值检测所述深度像素是否有效。Whether the depth pixel is valid is detected according to the confidence value and the confidence threshold. 2.根据权利要求1所述的检测方法,其特征在于,所述根据所述置信度值识别所述目标场景的类型,包括:2. The detection method according to claim 1, wherein the identifying the type of the target scene according to the confidence value comprises: 根据所有所述深度像素的所述置信度值生成置信度值直方图;generating a confidence value histogram according to the confidence values of all the depth pixels; 根据所述置信度值直方图识别所述目标场景的类型。The type of the target scene is identified according to the confidence value histogram. 3.根据权利要求1所述的检测方法,其特征在于,所述根据所述置信度值识别所述目标场景的类型,包括:3. The detection method according to claim 1, wherein the identifying the type of the target scene according to the confidence value comprises: 获取所述置信度值位于预设置信度值范围内的所述深度像素的数量占所有所述深度像素的数量的比例;及obtaining the ratio of the number of the depth pixels whose confidence value is within the range of the preset confidence value to the number of all the depth pixels; and 根据所述比例确定所述目标场景的类型。The type of the target scene is determined according to the scale. 4.根据权利要求1所述的检测方法,其特征在于,还包括:4. detection method according to claim 1, is characterized in that, also comprises: 根据预设标定信息和所述深度像素的图像坐标,修正所述置信度值。The confidence value is modified according to preset calibration information and the image coordinates of the depth pixels. 5.根据权利要求4所述的检测方法,其特征在于,还包括:5. detection method according to claim 4, is characterized in that, also comprises: 根据每个所述深度像素的所述深度值和修正后所述置信度值,确定每个所述深度像素的反射率;或者,According to the depth value of each of the depth pixels and the corrected confidence value, determine the reflectivity of each of the depth pixels; or, 根据每个所述深度像素的所述深度值、所述置信度值和图像坐标,确定每个所述深度像素的反射率。The reflectivity of each of the depth pixels is determined based on the depth value, the confidence value and the image coordinates of each of the depth pixels. 6.根据权利要求5所述的检测方法,其特征在于,所述根据所述目标场景的类型确定置信度阈值,包括:6. The detection method according to claim 5, wherein the determining a confidence threshold according to the type of the target scene comprises: 根据所述深度像素的反射率对应的预设阈值和所述目标场景的类型确定每个所述深度像素对应的所述置信度阈值。The confidence threshold corresponding to each depth pixel is determined according to a preset threshold corresponding to the reflectivity of the depth pixel and the type of the target scene. 7.根据权利要求1所述的检测方法,其特征在于,所述检测方法还包括:7. The detection method according to claim 1, wherein the detection method further comprises: 根据所述当前深度图像的前N帧的所述深度像素的所述深度值、所述置信度值和有效次数,生成历史帧,所述N为正整数,所述有效次数根据所述当前深度图像的前N帧中对应位置的所述深度像素的所述置信度值和所述置信度阈值的比较结果确定。A historical frame is generated according to the depth value, the confidence value and the valid times of the depth pixels of the previous N frames of the current depth image, where N is a positive integer, and the valid times is based on the current depth A comparison result of the confidence value of the depth pixel at the corresponding position in the first N frames of the image and the confidence threshold is determined. 8.根据权利要求7所述的检测方法,其特征在于,所述检测方法还包括:8. The detection method according to claim 7, wherein the detection method further comprises: 根据所述当前深度图像的所述深度值和所述置信度值,更新所述历史帧。The historical frame is updated according to the depth value and the confidence value of the current depth image. 9.根据权利要求8所述的检测方法,其特征在于,所述根据所述当前深度图像的所述深度值和所述置信度值,更新所述历史帧,包括:9. The detection method according to claim 8, wherein the updating the historical frame according to the depth value and the confidence value of the current depth image comprises: 在所述当前深度图像的所述深度像素的第一深度值和所述历史帧中对应位置的所述深度像素的第二深度值的差值大于第一预设差值阈值的情况下,根据所述第一深度值和所述第二深度值加权生成第三深度值,以替换所述第二深度值,所述第一深度值的权重大于所述第二深度值的权重,所述第一深度值的权重和所述第二深度值的权重根据所述第一深度值的权重和所述第二深度值的差值确定;和/或In the case that the difference between the first depth value of the depth pixel of the current depth image and the second depth value of the depth pixel at the corresponding position in the historical frame is greater than a first preset difference threshold value, according to The first depth value and the second depth value are weighted to generate a third depth value to replace the second depth value, the weight of the first depth value is greater than the weight of the second depth value, the The weight of a depth value and the weight of the second depth value are determined according to the difference between the weight of the first depth value and the second depth value; and/or 在所述当前深度图像的所述深度像素的第一置信度值和所述历史帧中对应位置的所述深度像素的第二置信度值的差值大于第二预设差值阈值的情况下,根据所述第一置信度值和所述第二置信度值加权生成第三置信度值,以替换所述第二置信度值,所述第一置信度值的权重大于所述第二置信度值的权重,所述第一置信度值的权重和所述第二置信度值的权重根据所述第一置信度值的权重和所述第二置信度值的差值确定。In the case that the difference between the first confidence value of the depth pixel of the current depth image and the second confidence value of the depth pixel at the corresponding position in the historical frame is greater than a second preset difference threshold , generating a third confidence value by weighting according to the first confidence value and the second confidence value to replace the second confidence value, the weight of the first confidence value is greater than that of the second confidence value The weight of the degree value, the weight of the first confidence degree value and the weight of the second confidence degree value are determined according to the difference between the weight of the first confidence degree value and the second confidence degree value. 10.根据权利要求7所述的检测方法,其特征在于,所述根据所述置信度值和所述置信度阈值检测所述深度像素是否有效,还包括:10. The detection method according to claim 7, wherein the detecting whether the depth pixel is valid according to the confidence value and the confidence threshold further comprises: 在当前深度像素在所述历史帧中位置对应的目标深度像素的有效次数大于预设次数、且所述当前深度像素的所述置信度值大于所述置信度阈值的情况下,确定所述当前深度像素有效;或者In the case that the valid times of the target depth pixel corresponding to the position of the current depth pixel in the historical frame is greater than a preset number of times, and the confidence value of the current depth pixel is greater than the confidence threshold, determine the current depth pixel. Depth pixels are valid; or 在所述当前深度像素在所述历史帧中位置对应的目标深度像素的有效次数大于预设次数、且所述当前深度像素的所述置信度值小于所述置信度阈值的情况下,使用所述目标深度像素的目标深度值和所述当前深度像素的当前深度值重新加权生成所述当前深度值,所述当前深度值的权值和所述目标深度值的权值根据所述当前深度像素和所述目标深度像素的深度值差值和置信度值差值中至少一个确定;或者In the case that the valid times of the target depth pixel corresponding to the position of the current depth pixel in the historical frame is greater than a preset number of times, and the confidence value of the current depth pixel is smaller than the confidence threshold, use the The target depth value of the target depth pixel and the current depth value of the current depth pixel are re-weighted to generate the current depth value, and the weight of the current depth value and the weight of the target depth value are based on the current depth pixel. Determine at least one of the depth value difference and the confidence value difference of the target depth pixel; or 在所述当前深度像素在所述历史帧中位置对应的目标深度像素的有效次数小于预设次数的情况下,确定所述当前深度像素无效;或者In the case that the valid times of the target depth pixel corresponding to the position of the current depth pixel in the historical frame is less than a preset number of times, it is determined that the current depth pixel is invalid; or 在所述当前深度像素在所述历史帧中位置对应的目标深度像素的有效次数小于预设次数、且所述当前深度像素的所述置信度值小于所述置信度阈值的情况下,确定所述当前深度像素无效。In the case that the valid times of the target depth pixel corresponding to the position of the current depth pixel in the historical frame is less than a preset number of times, and the confidence value of the current depth pixel is less than the confidence threshold, determine the The current depth pixel described is invalid. 11.根据权利要求1所述的检测方法,其特征在于,还包括:11. The detection method according to claim 1 , further comprising: 根据当前深度像素周围预设尺寸范围内的所述深度像素的所述深度值和所述置信度值,对所述当前深度像素进行滤波,以更新所述当前深度像素的所述深度值和所述置信度值。Filter the current depth pixel according to the depth value and the confidence value of the depth pixel within a preset size range around the current depth pixel to update the depth value and the confidence value of the current depth pixel the confidence value. 12.根据权利要求11所述的检测方法,其特征在于,所述根据当前深度像素周围预设尺寸范围内的所述深度像素的所述深度值和所述置信度值,对所述当前深度像素进行滤波,以更新所述当前深度像素的所述深度值和所述置信度值,包括:12 . The detection method according to claim 11 , wherein, according to the depth value and the confidence value of the depth pixel within a preset size range around the current depth pixel, the current depth pixel is determined. 12 . pixel filtering to update the depth value and the confidence value of the current depth pixel, including: 统计当前深度像素周围预设尺寸范围内的所述深度像素的所述置信度与所述当前深度像素的所述置信度的差值小于预定差值的目标深度像素的数量;Counting the number of target depth pixels in which the difference between the confidence level of the depth pixel and the confidence level of the current depth pixel within a preset size range around the current depth pixel is less than a predetermined difference value; 若所述数量大于第一预设数量,则根据所述当前深度像素的深度值和所述目标深度像素的深度值确定滤波深度值、并根据所述当前深度像素的置信度值和所述目标深度像素的置信度值确定滤波置信度值;If the number is greater than the first preset number, determine the filtering depth value according to the depth value of the current depth pixel and the depth value of the target depth pixel, and determine the filter depth value according to the confidence value of the current depth pixel and the target depth The confidence value of the depth pixel determines the filtering confidence value; 将所述当前深度像素周围预设尺寸范围内的所有所述深度像素的深度值均替换为所述滤波深度值、并将所述当前深度像素周围预设尺寸范围内的所有所述深度像素的置信度值均替换为所述滤波置信度值。Replace the depth values of all the depth pixels in the preset size range around the current depth pixel with the filtered depth value, and replace the depth values of all the depth pixels in the preset size range around the current depth pixel. The confidence values are all replaced with the filtered confidence values. 13.根据权利要求1所述的检测方法,其特征在于,所述根据所述置信度值和所述置信度阈值检测所述深度像素是否有效,还包括:13. The detection method according to claim 1, wherein the detecting whether the depth pixel is valid according to the confidence value and the confidence threshold further comprises: 统计当前深度像素周围预设尺寸范围内的所有所述深度像素中,置信度值大于所述置信度阈值的深度像素的数量;Counting the number of depth pixels whose confidence value is greater than the confidence threshold among all the depth pixels within a preset size range around the current depth pixel; 在所述数量小于第二预设数量的情况下,确定所述当前深度像素无效;In the case that the number is less than the second preset number, it is determined that the current depth pixel is invalid; 在所述数量大于第三预设数量的情况下,确定所述当前深度像素有效。When the number is greater than a third preset number, it is determined that the current depth pixel is valid. 14.根据权利要求12所述的检测方法,其特征在于,还包括:14. The detection method according to claim 12, characterized in that, further comprising: 确定所述可见光图像中与所述当前深度像素位置对应的目标可见光像素;determining a target visible light pixel corresponding to the current depth pixel position in the visible light image; 判断所述目标可见光像素周围所述预定尺寸范围内的可见光像素中,像素值与所述目标可见光像素的像素值的差值小于第四预设差值阈值的相似可见光像素;judging that among the visible light pixels within the predetermined size range around the target visible light pixel, the difference between the pixel value and the pixel value of the target visible light pixel is smaller than a similar visible light pixel with a fourth preset difference threshold; 根据所述相似可见光像素重新确定所述预设尺寸范围。The preset size range is re-determined according to the similar visible light pixels. 15.一种检测装置,其特征在于,所述装置包括:15. A detection device, characterized in that the device comprises: 拍摄模块,用于拍摄目标场景以获取深度图像,所述深度图像包括多个深度像素,每个所述深度像素的深度值均存在对应的置信度值,所述置信度值根据所述深度像素对应的光接收器接收的光量确定;a shooting module, configured to shoot a target scene to obtain a depth image, the depth image includes a plurality of depth pixels, and the depth value of each of the depth pixels has a corresponding confidence value, and the confidence value is based on the depth pixel The amount of light received by the corresponding optical receiver is determined; 识别模块,用于根据所述置信度值识别所述目标场景的类型;an identification module, configured to identify the type of the target scene according to the confidence value; 第一确定模块,用于根据所述目标场景的类型确定置信度阈值;a first determining module, configured to determine a confidence threshold according to the type of the target scene; 第二确定模块,用于根据所述置信度值和所述置信度阈值检测所述深度像素是否有效。The second determination module is configured to detect whether the depth pixel is valid according to the confidence value and the confidence threshold. 16.一种电子设备,其特征在于,包括处理器,所述处理器用于执行如权利要求1-14任一项所述的检测方法。16. An electronic device, characterized by comprising a processor for executing the detection method according to any one of claims 1-14. 17.一种计算机程序的非易失性计算机可读存储介质,其特征在于,当所述计算机程序被一个或多个处理器执行时,实现权利要求1-14中任一项所述的检测方法。17. A non-volatile computer-readable storage medium for a computer program, characterized in that, when the computer program is executed by one or more processors, the detection of any one of claims 1-14 is implemented method.
CN202210531141.0A 2022-05-16 2022-05-16 Detection method and device, electronic equipment and storage medium Pending CN114862804A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210531141.0A CN114862804A (en) 2022-05-16 2022-05-16 Detection method and device, electronic equipment and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210531141.0A CN114862804A (en) 2022-05-16 2022-05-16 Detection method and device, electronic equipment and storage medium

Publications (1)

Publication Number Publication Date
CN114862804A true CN114862804A (en) 2022-08-05

Family

ID=82636609

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210531141.0A Pending CN114862804A (en) 2022-05-16 2022-05-16 Detection method and device, electronic equipment and storage medium

Country Status (1)

Country Link
CN (1) CN114862804A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117197593A (en) * 2023-11-06 2023-12-08 天河超级计算淮海分中心 Medical image pseudo tag generation system

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107710734A (en) * 2015-07-24 2018-02-16 奥林巴斯株式会社 Image processing apparatus, image processing method and program
CN108810413A (en) * 2018-06-15 2018-11-13 Oppo广东移动通信有限公司 Image processing method and device, electronic equipment and computer readable storage medium
CN110378946A (en) * 2019-07-11 2019-10-25 Oppo广东移动通信有限公司 Depth map processing method and device and electronic equipment
TW202130973A (en) * 2019-09-27 2021-08-16 日商索尼半導體解決方案公司 Information processing device, correction method, and program

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107710734A (en) * 2015-07-24 2018-02-16 奥林巴斯株式会社 Image processing apparatus, image processing method and program
CN108810413A (en) * 2018-06-15 2018-11-13 Oppo广东移动通信有限公司 Image processing method and device, electronic equipment and computer readable storage medium
CN110378946A (en) * 2019-07-11 2019-10-25 Oppo广东移动通信有限公司 Depth map processing method and device and electronic equipment
TW202130973A (en) * 2019-09-27 2021-08-16 日商索尼半導體解決方案公司 Information processing device, correction method, and program

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117197593A (en) * 2023-11-06 2023-12-08 天河超级计算淮海分中心 Medical image pseudo tag generation system

Similar Documents

Publication Publication Date Title
KR102660109B1 (en) Method and apparatus for determining depth map for image
US10997696B2 (en) Image processing method, apparatus and device
CN110378946A (en) Depth map processing method and device and electronic equipment
CN110553665A (en) automatic measuring device and method for optical axis deviation of laser range finder
CN110400342B (en) Parameter adjustment method, device and electronic device for depth sensor
CN112367476B (en) Exposure time determining method and device of TOF camera and terminal equipment
CN113379847B (en) Abnormal pixel correction method and device
WO2023005057A1 (en) Camera cover cleanliness measuring method and measuring apparatus
CN115131215A (en) A kind of image correction method and screen system
CN114862804A (en) Detection method and device, electronic equipment and storage medium
CN112995525B (en) Camera exposure method and device for self-propelled equipment
CN113052886A (en) Method for acquiring depth information of double TOF cameras by adopting binocular principle
CN115830131A (en) Method, device and equipment for determining fixed phase deviation
CN112770100B (en) Image acquisition method, photographic device and computer readable storage medium
CN117572383B (en) A radar noise filtering method, device, host computer and laser radar
US10091469B2 (en) Image processing apparatus, image processing method, and storage medium
WO2021239029A1 (en) Method and device for compensating for phantom reflection
CN112752088B (en) Depth image generation method and device, reference image generation method and electronic equipment
CN119277035B (en) Projector backlight adjusting method, device, equipment and medium
US20250044451A1 (en) Enhancing depth representations based on amplitudes of time-of-flight signals
WO2024200575A1 (en) Object tracking circuitry and object tracking method
CN117872385A (en) Combined depth measuring device
CN118762672A (en) A method of adjustment
CN119559258A (en) A visible light near-infrared fusion positioning method and system based on feature consistency
CN117471683A (en) An adaptive hybrid variable-resolution ghost imaging method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination