JP2009139995A - Unit and program for real time pixel matching in stereo image pair - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a unit for accurately matching pixels corresponding to a stereo image pair in real time. <P>SOLUTION: The unit for matching the pixel pair of a corrected stereo image pair maps pixels in an image of a range camera to pixels in the right and left images of a stereo camera system. The unit includes a foreground/background mapping module 146 for comparing the pixel value of the pixel in the range camera image with a threshold, and in response to the foreground/background mapping module 146, a disparity retrieval module 148 for selectively performing a first disparity retrieval or a second disparity retrieval, in regard to the pixel in the second image corresponding to the pixel in the first image. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an apparatus and computer program for real-time disparity estimation of stereo images, and more particularly to real-time stereo image pairs for 2D / 3D head posture detection, recognition, 3D game development, animation, broadcast and communication. The present invention relates to an apparatus and a program for pixel matching for each frame.

  In media broadcast content, transmission and audio-video multimedia technologies are evolving at a fast pace to achieve a genuine feel. In order to make the viewer feel authentic, surreal audio and video content must be distributed to the viewer. The present invention focuses on acquiring three-dimensional (3D) visual content.

  Over the years, various types of 3D display technologies such as display based on the well-known red / green stereoscopic glasses, volume display (Non-Patent Document 1), auto-stereoscopic display (Non-Patent Documents 2 and 3) have been developed. . Display based on red / green stereoscopic glasses requires input from two cameras, but requires little processing. However, viewers must wear special glasses. Auto-stereoscopic 3D display does not require special glasses, but content must be captured in 3D and distributed in 3D. There is still considerable difficulty in reaching the quality of existing television broadcasts with the acquisition of content in 3D of natural scenes.

  In contrast to conventional 2D television, autostereoscopic 3D television requires not only the visual appearance of a scene but also the dense map information of that scene. For this reason, there are two main scientific challenges. That is, obtaining the visual appearance and depth map of the scene in real time and the resulting 3D appearance on the 3D television must match the original visual appearance of the scene (genuineness) ).

  Until now, many techniques have been proposed for 3D scene capture. In one extreme, the depth map is estimated by an image-based approach using a conventional calibrated stereoscopic camera. In another extreme example, the depth is obtained using an optical-based camera using time-of-flight (TOF) of infrared light.

  The TOF3D camera emits a sinusoidally modulated infrared light signal (Non-Patent Document 4). The time required for the movement of light until it exits the measurement system to the object and returns to the system is measured, and the depth of each pixel in the image is calculated. The best achievable distance accuracy is on the order of a few centimeters depending on the depth distance, with limited operating range, and measurements outside the range are unreliable. For this reason, the acquisition approach by the TDF camera is not suitable for 3D television broadcasting.

Conventional approaches based on calibrated stereoscopic vision (Non-Patent Documents 5 and 6) use an image processing algorithm, ie, template matching, to calculate a depth map of a scene from left and right camera images. This requires the camera to be calibrated to estimate internal and external camera parameters. The depth map estimation process includes template matching for synchronous capture, correction using calibration parameters, and disparity estimation between pixels of the left and right camera images. Template matching algorithms typically use a normalized cross-correlation based similarity measure.
B. Brandel and A.M. J. et al. Schwarz, “Classification of Volume Display Systems: Image Space Characteristics and Predictability”, IEEE Visualization and Computer Graphic Transactions, Vol. 8, No. 1, pp. 66-76, 2002 (B. Blundell and AJ Schwarz, “ The classification of volumetric display systems: Characteristics and predictability of the image space, "IEEE Transactions on Visualization and Computer Graphics, vol. 8, no. 1, pp. 66-76, 2002.) Philips Research, “Multi-view autostereoscopic display”, [online], search on December 3, 2007, Internet <URL: http://www.research.philips.com/technologies/display/ov_3ddisp.html> ( Philips Research, "Multi-view autostereoscopic displays," in http://www.research.philips.com/technologies /display/ov_3ddisp.html) R. Borner, B.B. Duxtine, O. Machiu, R.M. Ladder, T.W. Thinning and T.W. Sicola, “Family of single-user autostereoscopic display with head tracking capability” Circuit and System Transactions for IEEE Video Technology, Vol. 10, No. 2, pages 234-243, 2000 (R. Borner, B. Duckstein, O. Machui, R. Rder, T. Sinning, and T. Sikora, "A family of single-user autostereoscopic displays with head-tracking capabilities," IEEE Transactions on Circuits and Systems for Video Technology, vol 10, no. 2, pp. 234-243, 2000.) B. Butgen, T.W. Ojiya and M.M. Lehmann, “Ccd / cmos Lock-in Pixel for Range Imaging: Challenges, Limits and Current Status” Range Imaging Research Day 1, Proceedings 21-23, Ingen Sand / Carlmann (ed.), Zurich, 2005 (B. Buttgen, T. Oggier, and M. Lehmann, "Ccd / cmos lock-in pixel for range imaging: challenges, limitations and state-of-the-art," in Proceedings of 1st range imaging research day, pp. 21-32, Ingensand / Kahlmann (eds.), Zurich, 2005.) K. Konorige, “Small Vision Systems: Hardware and Implementation,” 8th International Symposium on Robotics Research, Hayama, Japan, 1997 (K. Konolige, “Small vision systems: Hardware and implementation,” in Eighth International Symposium on Robotics Research, (Hayama, Japan, 1997.) C. Sun, “Fast Stereo Matching Using Rectangular Sub-Regionization and 3d Maximum Area Technology”, Computer Vision International Journal, Vol. 47, 1/2/3, pages 99-117, 2002 (C. Sun, “Fast stereo matching using rectangular sub-regioning and 3d maximum-surface techniques, "International Journal of Computer Vision, vol. 47, no. 1/2/3, pp. 99-117, 2002.) G. R. Glymet and D.C. R. Stillzeka, Probability and Random Processing, 2nd edition, Clarendon Press, Oxford. ISBN 0-19-85366508, 1992 (GR Grimmett and DR Stirzaker, Probability and Random Processes, 2nd Edition, Clarendon Press, Oxford. ISBN 0-19-853665-8, 1992.) P. Viola and M.M. Jones, “Robust Real-Time Object Detection” Second International Workshop on Statistics and Theory of Visual Modeling, Learning, Calculation and Sampling, Vancouver, Canada, 2001 (P. Viola and M. Jones, “Robust real- time object detection, "in 2nd International Workshop on Statistical and Computational Theories of Vision-Modeling, Learning, Computing, and Sampling, Vancouver, Canada, 2001.) R. J. et al. Scharkov, Digital Image Processing and Computer Vision, John Wiley and Sons, 1989 (RJ Schalkoff, Digital Image Processing and Computer Vision, John Wiley and Sons, Inc., 1989.) T.A. Ojiya, R.D. Kaufman, M.C. Lehmann, P.A. Metzler, G. Lang, M.C. Schweizer, M.C. Richter, B.B. Butgen, N.C. Blank, K.K. Greasebach, B.B. Ullman, KH. Stageman and C.I. Elmer, "Real-time 3d imaging with a miniaturized optical range camera" Nuremberg Optical Conference, DE 2004 (T Oggier, R. Kaufmann, M. Lehmann, P. Metzler, G. Lang, M. Schweizer, M. Richter, B. Bttgen, N. Blanc, K. Griesbach, B. Uhlmann, K.-H. Stegemann, and C. Ellmers, "3d-imaging in real-time with miniaturized optical range camera," in Opto Conference Nurnberg, DE. , 2004.)

  However, traditional stereo-based 3D reconstruction approaches often cannot accurately calculate 3D information. This is because, depending on the location of the pixel in the stereo image, unique information for corresponding matching may not be provided, particularly when the search range for the stereo disparity search algorithm is unknown. Furthermore, template matching algorithms often cannot accurately calculate disparity when the image is noisy.

  Accordingly, one of the objects of the present invention is to provide an apparatus for accurately matching corresponding pixels of a stereo image pair in real time.

  Another object of the present invention is to provide an apparatus for accurately and robustly matching corresponding pixels of a stereo image pair in real time.

  Yet another object of the present invention is to provide an apparatus that improves the pixel matching of a stereo image pair by accurately matching the template of the left image with the corresponding block of the right image.

  A first aspect of the present invention relates to an apparatus for matching corresponding pixel pairs in a corrected stereo image pair. This apparatus is connected to a stereo camera and a range camera, and can receive a corrected stereo image pair and a range camera image, respectively. The stereo image pair includes a first image and a second image. The apparatus comprises: a first mapping means for mapping a pixel in the range camera image to a pixel in the first image and a pixel in the second image; and the pixel in the range camera image A comparison means for comparing the intensity of the first and second disparity searches in response to the comparison result by the comparison means in response to the comparison means And searching for pixels in the second image that match the pixels in the first image.

  Preferably, the search means is a first search means that responds when the comparison means determines that the intensity of the pixel is higher than the threshold value, and the comparison means determines that the intensity of the pixel is the value of the pixel. And a second search means for responding to the determination that it is equal to or less than the threshold value. The search range of the first search means is shorter than the search range of the second search means.

  More preferably, in response to the comparison unit determining that the intensity of the pixel is higher than the threshold value, the first search unit determines the pixel of the range camera image to be the second A second mapping means for mapping to an image; and a first search range extending on both sides of the pixel on the epipolar line of the second image mapped by the second mapping means, One search range defining means; a predetermined block of a predetermined size including the pixels in the first image; and the second including each pixel in the first search range. A first similarity calculating means for calculating a predetermined similarity measure with a block of a predetermined size in the image of the image, and the highest similarity among the blocks in the second image A first block selecting means for selecting one having a degree, and a pixel at the center of the block selected by the first block selecting means is selected as one that matches the pixel of the first image First pixel selection means. Here, “include” means that the block includes a pixel at a predetermined position, for example, a pixel at the center of the block if the block is rectangular.

  More preferably, the second search means includes second search range defining means for defining a second search range that is longer than the first search range, wherein the second search range is the first search range. Extending to only one side of the pixel on the epipolar line of the second image mapped by the second mapping means, the second search means further including the pixel in the first image in advance A predetermined similarity measure between a predetermined block of a predetermined size and a block of a predetermined size in the second image that includes each pixel in the second search range A second similarity calculating means for calculating, and a second for selecting the block in the second image having the highest similarity measure calculated by the second similar means. Block selection And a second pixel selection means for selecting a central pixel of the block selected by the second block selection means as a match with the pixel of the first image. .

  The first similarity calculation means includes means for dividing each of the block in the first image and the block in the second image into a plurality of sub-blocks having the same shape; Means for calculating an average pixel value of each of the blocks in the second image and the block of the first image; and calculating the average pixel value from the pixel values of each pixel of the sub-block. Means for subtracting, means for calculating an average pixel value of each pixel of the sub-block, an average pixel value of the sub-block of the first image, and each of the blocks of the second image Means for calculating a sum of squared errors with each of the average pixel values of the sub-blocks. The sum of the square errors is the similarity measure.

  A second aspect of the present invention relates to a computer-executable computer program that is connected to a stereo camera and a range camera and can receive a corrected stereo image pair and a range camera image. The stereo image pair includes a first image and a second image. The computer program, when executed on the computer, causes the computer to map a pixel in the range camera image to a pixel in the first image; A comparison means for comparing the pixel value of the pixel with a threshold value, and a first disparity search and a second disparity search in response to the comparison result by the comparison means in response to the comparison means Are selectively operated to search for pixels in the second image that match the pixels in the first image.

Depth Map Estimation Algorithm Stereo camera-based depth map estimation algorithms often rely on correction and search mechanisms. FIG. 7 represents a corrected stereo image pair template search process for disparity estimation. The rectangular template window 252 including the target point 250 (x, y) in the left camera image 240L is the same on the epipolar line 242R at the same height as the epipolar line 242L in the left camera image 240L in the right camera image 240R. The size block 262 is compared with various similarity measures such as square error or zero mean normalized correlation. The block with the highest similarity measure defines the pixel 260 (x + u, y) in the right camera image 240R. The pixel 260 and the pixel 250 in the left image are considered to be images of the same target point.

  The matching algorithm needs to calculate a similarity measure for the 2D block B262 of size w × h in the right camera image 240R of the 2D template window A252 in the left camera image 240L. In general, in the presence of correction error and Gaussian noise, A and B can be expressed as follows:

ここでθは、補正誤差による、ウィンドウデータＡに対するウィンドウデータＢの相対的回転である。γ及びΓはそれぞれ左及び右カメラの任意のスケール値（利得要素）を表す。Ｘ―（ここで―は式中文字の上に付されるものである）は平均値を表し、Ｘ〜（ここで〜は式中文字の上に付されるものである）はウィンドウデータの固有の形状又はテクスチャ特性を表すものである。Ｎ（μ,σ^２）は、平均μ及び分散σ^２≧０の時の任意のガウスノイズを表し、ここで画像とノイズデータとは独立である。すなわち、一般に、理想的な条件下では、ウィンドウデータの両方が同じ対象パッチに属し、θがゼロであり、ステレオ画像が完璧に補正されたときのみ、以下が成立する。

Here, θ is a relative rotation of the window data B with respect to the window data A due to a correction error. γ and Γ represent arbitrary scale values (gain factors) of the left and right cameras, respectively. X- (where-is attached to the letter in the formula) represents an average value, and X to (where ~ is attached to the letter in the formula) are window data. It represents a unique shape or texture characteristic. N (μ, σ ² ) represents an arbitrary Gaussian noise when the mean μ and the variance σ ² ≧ 0, where the image and the noise data are independent. In other words, generally, under ideal conditions, both of the window data belong to the same target patch, θ is zero, and the following holds only when the stereo image is perfectly corrected.

しかし、一般に、我々がアクセスできるのは式１及び式２のＡ及びＢの観察のみであり、画像は、較正誤差、映像ノイズ、又はＡ〜及びＢ〜が独自の情報を担持していない場合に独自の固有形状特性がないことにより、画像を完璧にマッチングすることはできないであろう。映像ノイズは、レンズ焦点の違い、視野角の違い、及び左右のカメラ上での照明効果のむら等のために生ずる。

However, in general, we can only access the observations of A and B in Equations 1 and 2, and the image is calibration error, video noise, or if A ~ and B ~ do not carry their own information Due to the lack of unique intrinsic shape characteristics, the image may not be perfectly matched. Video noise occurs due to differences in lens focus, differences in viewing angle, and uneven lighting effects on the left and right cameras.

TOF Camera In this embodiment, in order to reduce the number of failures or matching errors, the 3D measurements of the TOF camera are backprojected on-the-fly to both the left and right camera images 240L and 240R, so Estimate the location. Therefore, the search range of the disparity algorithm is limited to each pixel. The restriction algorithm will be described later.

  The TOF camera emits amplitude-modulated invisible infrared light that is reflected by the scene object and backscattered onto the image sensor. Each pixel on the image sensor demodulates the incoming optical signal and recovers the sine wave function to estimate the phase delay. In this case, the phase delay is directly proportional to the distance of the object to the camera (Non-Patent Document 7).

  From a practical point of view, TOF cameras have various problems and limitations. For example, the standard deviation of distance measurement behaves inversely proportional to the modulation frequency (Non-Patent Document 4). For this reason, a high modulation frequency is preferred when aiming at accurate distance measurement. On the other hand, the unique measurement range is inversely proportional to the modulation frequency. For example, when the modulation frequency is 20 MHz, the unique measurement range is 7.5 meters. For this reason, the measured values belonging to the ambiguous range (above 7.5 meters) are estimated inaccurately.

  Since the emitted infrared light follows the law of inverse square, the pixel with noise can be filtered out by setting the amplitude threshold value of the incoming pixel value. Basically, this can be thought of as having a range measurement for the foreground object but not for the background object. Therefore, in this embodiment, this information is used for masking the foreground and background areas in the conventional stereo reconstruction process.

Range Camera to Stereo Camera Mapping A set of 3D measurements from the TOF camera and their corresponding image points in the stereo image give the range camera to stereo camera mapping parameter A = [a _ij ]. The mapping equation from the range camera to the stereo camera for each camera in homogeneous coordinates can be written as the following well-known camera calibration procedure (Non-Patent Document 8).

ここで［ｘ_０ｙ_０ｚ_ｏ］^ＴはＴＯＦカメラからの３Ｄ測定値であり、［ｘ_ｉｙ_ｉ］^Ｔはその対応の画像点であり、ｗ_ｉは非ゼロのファクタである。ａが以下のようなベクトルの形のマッピングパラメータの組を表すものとする。

Where [x ₀ y ₀ z _o ] ^T is the 3D measurement from the TOF camera, [x _i y _i ] ^T is its corresponding image point, and w _i is a non-zero factor. Let a denote a set of mapping parameters in the form of a vector:

_{a = [a 11 a 12 a} 13 a 14 a 21 a 22 a 23 a 24 a 31 a 32 a 33] T.
By setting a ₃₄ = 1.0, Equation 3 can be scaled, and with some algebraic operations, two corresponding 3D-2D point pairs are linear in the a _ij parameter. Gives an expression. Ie

上の式を以下のように定式化することができる。

The above equation can be formulated as follows:

こうして、３Ｄ−２Ｄの点の対１つから２つの式と１１の未知数が与えられ、ここでＱはその行が式３の形である２×１１の行列であり、ａは式４の未知のマッピングパラメータのベクトルである。式３中の２つの式は線形独立でなければならないので、ここから、同一平面上にないマッピング点の最小数がｎ＝６であることがわかる。実際には、ｎ＞＞６を選択してＱ−Ｒ分解により式４を解く。上述の処理の結果、レンジカメラからステレオカメラへの２個のマッピング行列Ａ_Ｌ及びＡ_Ｒを、左右のステレオカメラそれぞれについてオフラインで得ることができる。図１５はＴＯＦカメラからステレオ画像へのマッピングの画面をキャプチャしたものであって、マッピングされた画素が左右のカメラ画像に重ねられている。

Thus, from one pair of 3D-2D points, two equations and 11 unknowns are given, where Q is a 2 × 11 matrix whose rows are in the form of Equation 3 and a is the unknown of Equation 4. This is a vector of mapping parameters. Since the two equations in Equation 3 must be linearly independent, it can be seen from this that the minimum number of mapping points that are not on the same plane is n = 6. In practice, n >> 6 is selected and Equation 4 is solved by QR decomposition. As a result of the above processing, two mapping matrices A _L and A _R from the range camera to the stereo camera can be obtained offline for each of the left and right stereo cameras. FIG. 15 shows a screen shot of mapping from a TOF camera to a stereo image. The mapped pixels are superimposed on the left and right camera images.

  I want to map 3D range data captured by a TOF camera to left and right stereo images on the fly so that the disparity search range can be limited to each pixel pair. Since wi ≠ 0 for any 3D-2D pair, Equation 3 can be rewritten as:

ここで［ｘ_０ｙ_０ｚ_０］^ＴはＴＯＦカメラからの画素の場所についての３Ｄ測定値であり、以下はそれぞれ、これに対応する左右のステレオカメラ画像の画像座標値である。

Here, [x ₀ y ₀ z ₀ ] ^T is a 3D measurement value for the location of the pixel from the TOF camera, and the following are the image coordinate values of the left and right stereo camera images corresponding thereto.

ＴＯＦカメラ自体に誤差があるので、右カメラ上で実際に対応する画素の場所は以下の通りとなり、ここでステレオカメラ画像は補正されており、δは前景対象物に対するあらたな探索範囲を表す定数（この実現例では８）である。

Since there is an error in the TOF camera itself, the location of the actually corresponding pixel on the right camera is as follows, where the stereo camera image has been corrected, and δ is a constant representing the new search range for the foreground object (8 in this implementation).

テンプレートマッチングのための正規化された相互相関類似尺度
立体視ディスパリティ探索アルゴリズム等の様々なパターンマッチングアルゴリズムにおいて、２個の画像ブロック間の類似度を測定するのに、相互相関値が利用される。これは、第２のカメラ画像内の一組のブロックを、一つづつ第１のカメラ画像からの既知のものと比較することによって、その一組のブロックのうちからマッチするものを見出すのに慣用される。ブロックのＤ．Ｃ．値は形状情報を担持していないので、式１及び式２から各ブロックの平均を除去する。これによって以下が得られる。

Normalized cross-correlation similarity measure for template matching Cross-correlation values are used to measure the similarity between two image blocks in various pattern matching algorithms such as stereoscopic disparity search algorithm . This can be done by comparing a set of blocks in the second camera image to the known ones from the first camera image one by one to find a match from the set of blocks. It is commonly used. D. of the block. C. Since the value does not carry shape information, the average of each block is removed from Equation 1 and Equation 2. This gives the following:

式６及び式７の個々の分散はウィンドウデータ及びノイズデータの分散の和に等しい。というのも、画像データ（あるシーンの視覚的外観）と画像ノイズとは独立だからである。したがって、分散は以下のように表すことができる。

The individual variances in Equations 6 and 7 are equal to the sum of the variances of window data and noise data. This is because image data (visual appearance of a scene) and image noise are independent. Thus, the variance can be expressed as:

そうでなければ、Ａ〜とＢ〜とはなんら独自の形状情報を担持しない。式６と式７との正規化相互相関式は次のように書くことができる。

Otherwise, A ~ and B ~ do not carry any unique shape information. The normalized cross-correlation equation between Equation 6 and Equation 7 can be written as

ここでｕは第２の画像のエピポーラ線上の場所（ｘ＋ｕ）に対応するウィンドウＡの前もって仮定された変位値であり、θは一般に、画像が補正されている場合は０であると仮定される。従って、以下の式ではパラメータθを無視することができる。

Where u is the pre-assumed displacement value of window A corresponding to the location (x + u) on the epipolar line of the second image, and θ is generally assumed to be zero if the image has been corrected. . Therefore, the parameter θ can be ignored in the following equation.

  To better analyze Equation 10, substituting Equation 6-7 and Equation 8-9 into Equation 10 and expanding it yields:

ここでＮ（０，σ^２）（ｘ,ｙ）は、Ｎ（０，σ^２）分布からくる画素の場所（ｘ,ｙ）でのノイズデータを表す。

Here, N (0, σ ² ) (x, y) represents noise data at the pixel location (x, y) coming from the N (0, σ ² ) distribution.

  Obviously, the reliability of the similarity measure given by Equation 11 is low when there is noise in the image. Therefore, to reduce false matching, noise effects must be removed from the similarity measure in the pattern matching task.

In the following, a new noise robustness measure (NRSM) algorithm for the template matching task will be described. The block data A ^ and B ^ (where the sign of ^ is attached to the letter in the formula) in Expression 6 and Expression 7 is divided into sub-blocks. This is as shown in FIG. 13 and is expressed below.

なぜなら、これらブロックの強度値は平均からの偏差だからである。各サブブロックを次の式に従ってシーンとノイズとに分解することができる。

This is because the intensity values of these blocks are deviations from the average. Each sub-block can be decomposed into scene and noise according to the following equation:

新たな係数の組を以下のように規定することによって、上述のノイズ項を除去する。

The above noise term is removed by defining a new set of coefficients as follows.

これによって次の式が得られる。

This gives the following equation:

a = [a ₁ , a ₂ ,. . . , A _N ], (15)
b = [b ₁ , b ₂ ,. . . , B _N ], (16)
In Equations 6 and 7, the noise term N (0, σ ² ) comes from independent and identically distributed (i.d.) random variables. Therefore, avg (A ^ _k-noise ) → 0 is established according to the law of large numbers. On the other hand, the following scene term avg (A ^ _k-scene ) comes from the scene pattern and does not necessarily approximate to zero due to its shape characteristics, but for unique matching with the template, At least one sub-block needs to produce a non-zero average value. Otherwise, it is considered that shape information is not available.

  For the above-described processing, the algorithm calculates an intermediate representation of the input image called “integrated image” (Non-Patent Document 9) for calculating a circle at a very high speed without depending on the block size. Briefly, an integrated image is obtained by summing the pixel values in an arbitrarily sized rectangle that encompasses the pixel for each pixel of the image and assigning the resulting sum to that pixel.

Now, a _k and b _k are released from the influence of noise, but are not released from the left and right camera scale factors, γ and Γ, respectively. To remove the scale factor, a new set of normalization descriptors will be defined as follows:

ここでａ_ｋ＞０，ｂ_ｋ＞０であり、かつ１≦ｋ≦Ｎである。最大の絶対値係数、ａ_ｋ及びｂ_ｋを正規化のために選択することで、（もし残っていても）正規化記述子上のノイズ効果をさらに低減することができる。その後、差分平方和(ｓｕｍ ｏｆ ｓｑｕａｒｅｄ ｄｉｆｆｅｒｅｎｃｅｓ ｍｅｔｈｏｄ）の方法を用いて、以下のように正規化記述子の組を比較する。

Here, a _k > 0, b _k > 0, and 1 ≦ k ≦ N. Choosing the largest absolute value coefficients, a _k and b _k for normalization can further reduce the noise effects on the normalization descriptor (if any). Then, using the sum of squared differences method, the set of normalized descriptors is compared as follows:

正規化相互相関ベースの類似尺度アルゴリズムが生成する結果は、ノイズがある場合には信頼性が低くなることを、数学的に示した。他方で、提案されたＮＲＳＭアルゴリズムは、画像のノイズ効果を低減又は除去する。予備的な実験結果は、理論が実践と一致することを示した。

The results generated by the normalized cross-correlation-based similarity measure algorithm have been shown to be less reliable in the presence of noise. On the other hand, the proposed NRSM algorithm reduces or eliminates the noise effect of the image. Preliminary experimental results showed that the theory is consistent with practice.

System settings [Structure]
As described above, this embodiment uses a stereoscopic-based disparity search algorithm whose performance is improved by the TOF camera in order to acquire real-time disparity data. In this embodiment, the search domain of the disparity algorithm is limited and the disparity search range is selected using the 3D location information of each pixel in the observed TOF camera image.

  FIG. 1 is a diagram showing the overall structure of the disparity estimation system 40 of this embodiment. Referring to FIG. 1, a disparity estimation system 40 includes a camera assembly 52 including a stereo camera 60 and a TOF camera 62, a 3D monitor 54, a normal monitor 74, a stereo image stream from the stereo camera 60, A real-time disparity calculating device 50 that calculates the disparity of each pixel pair of the left and right camera images from the TOF image stream from the TOF camera 62. The 3D monitor 54 can convert from 2D to 3D. That is, given a 2D image stream and a corresponding disparity image stream, the 3D monitor 54 calculates a right image based on the given image. An inclined lenticular lens is placed on the display screen of the 3D monitor 54, and the 3D monitor 54 focuses the light from the different pixels in a pre-defined direction so that the viewer sees different sides of the picture. The left and right images are displayed alternately as shown.

  FIG. 2 is a hardware block diagram of the real-time disparity calculating apparatus 50 of this embodiment implemented by a computer. Referring to FIG. 2, disparity calculating apparatus 50 includes a computer 70, a mouse 82, and a keyboard 80 in addition to a 3D monitor 54 and a monitor 74. The mouse 82, the keyboard 80, and the monitors 54 and 74 are all connected to the computer 70.

  Still referring to FIG. 2, the computer 70 includes a central processing unit (CPU) 90, a bidirectional data and address bus 92 connected to the CPU 90, and a read-only memory (Read) connected to the bus 92. In order to drive only memory (ROM) 94, random access memory (RAM) 96 connected to bus 92, hard disk drive 98 connected to bus 92, and DVD medium 108 connected to bus 92. Digital versatile disc (DVD) drive 128 and a bus 92 connected to receive a stereo image stream from stereo camera 60, and a TF from TOF camera 62. A video capture board 102 for receiving the OF image stream, a semiconductor memory drive 106 connected to the bus 92 for driving the semiconductor memory 110, and a graphics processing unit (connected to the bus 92, the 3D monitor 54 and the monitor 74). (Graphic Processing Unit: GPU) 104. All of these components of computer 70 are connected to bus 92 and are mutually accessible.

  In another aspect, the function of the real-time disparity calculation apparatus 50 is realized by software executed on the computer 70. FIG. 3 is a functional block diagram of the real-time disparity calculating apparatus 50.

  Referring to FIG. 3, functionally, the real-time disparity calculator 50 includes a calibration frame memory 138 and a stereo camera 60 for storing images from the TOF camera 62 during preliminary calibration of the TOF camera 62. Calibration software 130 for calibrating and outputting calibration parameters of the stereo camera 60, calibration parameter memory 132 for storing the calibration parameters output by the calibration software 130, and calibration parameters stored in the calibration parameter memory 132 Utilizing correction software 134 for correcting a stereo image pair from the stereo camera 60 and a frame memory 136 for storing left and right images output by the correction software 134. During calibration, calibration software 130 reads the images stored in frame memory 136 that have not been corrected by correction software 134 and calculates calibration parameters for stereo camera 60.

  The real-time disparity calculation device 50 further maps the foreground object to map the TOF camera measurement values stored in the calibration frame memory 138 to the left and right images corrected by the correction software 134 and stored in the frame memory 136. Pre-calibration software 140 for calculating the matrix, TOF mapping parameter memory 142 for storing the parameters of the mapping matrix calculated by the pre-calibration software 140, and for each pixel of the TOF image in the calibration frame memory 138, the TOF mapping parameter memory 142 Is used to calculate the location of the corresponding 2D pixel in the left and right images, and is likely to be high if the pixel value of the selected TOF pixel is higher than a predetermined pixel value threshold. If the foreground / background mapping module 146 and the foreground / background mapping module 146 for outputting the foreground / background (F / G) signal 176 to be low level, And a threshold storage unit 144 for storing.

  As described above, the infrared light emitted from the TOF camera 62 follows the inverse square law. Therefore, by setting an incoming threshold value, it is possible to filter out pixels including noise in the background (far from the TOF camera 62).

  The real-time disparity calculation device 50 is further connected to receive the F / G signal 176 and the image stored in the frame memory 136, and searches for a pixel in the right camera image corresponding to each pixel in the left camera image. The disparity search module 148 for calculating the disparity between the left and right pixels, and the disparity calculated by the disparity search module 148 are stored at addresses corresponding to the pixels of the left camera image. Select the disparity memory 150, the calibration frame memory 138, the image stored in the frame memory 136, the disparity image stored in the disparity memory 150, or any combination of these images for the 3D monitor 54 and the monitor 74 Graphic output And a Tsu door 152. Given a stream of a 2D image of the left camera and its corresponding disparity image, the 3D monitor 54 converts the 2D image and disparity image into a 3D image stream and displays it.

  The real-time disparity calculation apparatus 50 further includes a controller 122 that controls the overall operation of the modules in the real-time disparity calculation apparatus 50. The functions of the preliminary calibration software 140, the calibration software 130, the correction software 134, the foreground / background mapping module 146, the disparity search module 148, and the controller 122 are all implemented by software executed on the computer 70. Although not shown in FIG. 3, the software and modules of these components communicate with the controller 122 and utilize the preferred GUI on the monitor 74 in operation. The function of the graphic output unit 152 is realized by a combination of software executed by the CPU 90 and the CPU 104.

In this embodiment, the calibration software 130 is used offline to calculate the calibration parameters A ₁ and A ₂ of the stereo camera 60. Calibration is performed to correct radial distortion, lens eccentricity, focal length, pixel aspect ratio, baseline, and orientation of each of the cameras 60L and 60R. Calibration parameters are stored in calibration parameter memory 132. In the calibration process of this embodiment, the user presents a predetermined pattern to the stereo camera 60. The calibration software 130 uses the output of the stereo camera 60 to calculate parameters. Software for calibration is commercially available. For example, Small Vision System ^™ (SVS ^™ ) software distributed by SRI International can be used.

  The correction software 134 is used to correct the output stereo image of the stereo camera 60. Here, the correction is to align the corresponding epipolar lines of the left and right images from the stereo camera 60. This process is shown in FIG.

  Referring to FIG. 6, it is assumed that left and right images 230L and 230R of stereo camera 60 include a line 232L and a corresponding line 232R, respectively. Due to the difference between the lens distortion and the lens orientation, the left and right images have different shapes at different positions even if the parallax is different.

  By correcting the image, the corresponding lines 232L and 232R are aligned with the rows of the image, resulting in calibrated epipolar lines 242L and 242R of the left and right camera images 240L and 240R. Without correction, real-time disparity search is almost impossible. The correction is performed by a predetermined calculation. In this calculation, the calibration parameters stored in the calibration parameter memory 132 are used. Correction software is also commercially available.

  In normal operation, the image from the stereo camera 60 is corrected by the correction software 134 and stored in the frame memory 136.

  Pre-calibration software 140 is used offline for pre-calibration of the image from TOF camera 62 and the corrected image of stereo camera 60. This process will be described with reference to FIGS.

  A graphical user interface is implemented to estimate the mapping matrix from the TOF camera 62 to a stereo image. The user presents a predetermined pattern 340 shown in FIG. 11 to the TOF camera 62 and the stereo camera 60. There are a plurality of markers 344A, 344B, 344C, 334D and 344E on the surface of the pattern 340. With reference to FIG. 12A, an image 360 of a pattern 340 on the TOF camera 62 is first displayed on the monitor 74. The user places the pointer 370 on one of the markers and clicks the mouse button. The xy coordinates of the clicked position are stored in the memory. Next, an image 362L of the pattern 340 in the left camera image is displayed on the monitor 74. As shown in FIG. 12B, the user places the pointer 370 on the same marker and clicks the mouse button. The xy coordinates of the clicked position are stored in the memory. Similarly, as shown in FIG. 12C, the same point-click operation is performed on the image 362R of the pattern 340 of the right camera image, and the xy coordinate values are stored in the memory.

Thus, the graphical user interface allows the user to select a triple of n (n >> 6) target image points off-line, and Equation 4 is solved by QR decomposition. As a result, two mapping matrices A _L and A _R are obtained for the left and right stereo cameras, respectively. Matrix A _L is the pixel in the TOF camera image is used to map to the corresponding pixel of the left camera image. The matrix _AR is used to map pixels to corresponding pixels in the right camera image. Parameters defining the A _L and _{A R} are stored in the TOF mapping parameter memory 142.

  During video capture, TOF camera measurements of foreground objects are mapped to stereo images, limiting the search range of the stereo disparity algorithm on the fly.

FIG. 4 shows the overall structure of the foreground / background mapping module 146. Referring to FIG. 4, the foreground / background mapping module 146 is connected to receive a pixel selection unit 170 that selects pixels in order from the upper left to the lower right of the TOF camera image, and to receive the output of the pixel selection unit 170. A pixel reading unit 172 for reading a pixel value (intensity) from a pixel in the image of the TOF camera 62 selected by the selection unit 170 and stored in the calibration frame memory 138, and a parameter stored in the TOF mapping parameter memory 142 Is used to map the xy coordinate values of the selected pixel to the coordinate values x _L and y _L in the left camera image respectively by applying the mapping matrix A _L to x and y respectively. The xy coordinates of the selected pixel by applying the part 174 and the mapping matrix _AR to x and y respectively A right mapping unit 178 for mapping values to coordinate values x _R and y _R in the right camera image, a pixel value of the selected pixel from the pixel reading unit 172, and a threshold value from the threshold storage unit 144 And a comparator 180 connected to receive the intensity and for outputting an F / G signal 176. As described above, the F / G signal 176 is at the high level if the intensity of the pixel selected by the pixel selection unit 170 is higher than the threshold value, and is at the low level otherwise.

FIG. 5 shows the overall structure of the disparity search module 148 shown in FIG. Referring to FIG. 5, the disparity search module 148 is connected to receive x _L and y _L from the foreground / background mapping module 146 and stores (x _L , y _{L L} ) of the left image stored in the frame memory 136. ) Having a left block readout module 200 for reading out pixels, an input connected to receive the F / G signal 176, connected to the output of the left block readout module 200, and two outputs 202a and 202b. The selection unit 202 for selectively outputting the pixel value received from the left block readout module 200 to either the output 202a or 202b depending on the level of the F / G signal 176, and the input of the selection unit 202 Connected to the output 202a and corresponding to a pixel in the right camera image that is read from the left block readout module 200 A foreground disparity search module 208 for searching for one using a foreground disparity search algorithm and outputting a disparity value between left and right pixels, and a left block readout module 200 for pixels in the right camera image. A background disparity search module 210 for searching for a pixel corresponding to the left pixel read out from the image using a background disparity search algorithm and outputting a disparity value between the left and right pixels; A selection unit 212 that selects one of the outputs of the foreground disparity search module 208 and the background disparity search module 210 depending on the level. The output of the selection unit 212 is connected to the data input port of the disparity memory 150.

  The disparity search module further includes a foreground interval memory 204 that stores a length of an interval used for disparity search in the foreground disparity search algorithm, and an interval length used for disparity search in the background disparity search algorithm. 3 to control the disparity memory 150 shown in FIG. 3 so that the disparity value selected by the background interval memory 206 and the selection unit 212 is stored at the respective left pixel addresses of the disparity memory 150. A disparity memory control unit 214. When the calculation of the disparity of a certain pixel is completed, either the foreground disparity search module 208 or the background disparity search module 210 outputs a signal indicating the end of the search, which is selected by the selection unit 212, and foreground / background mapping. The pixel selection unit 170 of the module 146 is provided, and thus the pixel selection unit 170 selects the next pixel.

  Referring again to FIG. 7, the disparity search algorithm generally includes the following steps. That is, selecting a pixel 250 (x, y) in the left camera image 240L, defining a rectangular template window 250 including the target pixel 250 (x, y) in the left camera image 240L, and the right camera Selecting (x + u, y) pixel 260 in image 240R and using the similarity measure described above, a block of the same size including pixel 260 at its center on epipolar line 242R in right camera image 240R 262 and the rectangular template window 252 are compared; a value u is changed within a predetermined interval to calculate a similarity measure with the block 262 of the same size as the rectangular template window 252; Measure while changing until it moves from the leftmost position to the rightmost position within a set interval. Comprising the steps of repeating the steps of, selecting a block having the highest similarity measure in right camera image 240R, and selecting the pixels 260 in the selected block as the pixel corresponding to the pixel 250.

  Referring to FIG. 8, the distance between the pixel 250L in the left camera image 240L and the corresponding pixel 250R in the right camera image 240R defines the disparity D.

  If the object is in the foreground, the disparity between its left and right camera images should be greater than that of the symmetrical object in the background. Therefore, the search interval must be increased. If the search interval becomes longer, the time required for the search also becomes longer, and disparity search in real time becomes difficult.

However, the time required for calculating the disparity in the disparity estimation system 40 of this embodiment is considerably short. This is because the disparity estimation system 40 uses the TOF camera 62. Referring to FIGS. 9 (A) and 9 FIG. 9 (B), the pixels in the TOF camera image is mapped to the pixel 250 _L in the left camera image 240 _L is selected by the left mapping matrix _{A L.} Based on the pixel value of the selected pixel in the TOF camera image, it is determined whether the pixel belongs to the foreground object.

If the pixel belongs to the foreground object, the coordinate value of the corresponding pixel 260R of the right camera image 240R is estimated using the mapping matrix A _R. A search interval 270 is defined so as to include the estimated pixel 260R. In general, the x-coordinate values of the pixels 250L and 260R are different from each other. These are shown respectively as the distance _{D L} and _{D R} in FIG. 9 (A). In this embodiment, the search interval 270 is selected such that the estimated pixel 260R is in the center of the search interval 270. In other expressions, if if the pixel 250L of x-axis coordinates x _L, corresponding pixels in the right camera image 240R is searched in intervals of _{[x R -δ, x R +} δ], where [delta] is a constant ( a 8) in this implementation, x _R is the x-coordinate of the pixels mapped by a matrix AR.

When the pixel belongs to the object in the background, the pixel 260R at the same position as the target pixel 250L is selected in the right camera image 240R. In this case, the search interval 272 is defined as an interval 272 on the right side of the pixel 260R. In other words, if x _L is the x-axis coordinate value of the left camera image, the location of the actual corresponding pixel of the right camera may be searched at intervals of [x _L , x _L + L], where The stereo camera image is corrected, and L is a constant (40 in this implementation) representing the search range of the background object.

  Because of the mapping matrix that maps the pixels of the TOF camera image to the stereo image, the corresponding pixels in the left and right images are roughly known, so the search range is limited. Further, since it is known whether or not the pixel belongs to the foreground object, a search algorithm suitable for the foreground or background object pixel can be selected. A preferred search range is predefined for each of the algorithms. In particular, the search interval for the background pixels is considerably limited compared to that of the foreground pixels, so the calculation cost is low and the possibility of incorrect matching is very low. Therefore, errors (if any) can be corrected in calculating disparity.

  Note that the 3D X and Y coordinate values have no effect on the disparity search range. Only the Z coordinate is affected. The Z coordinate value is roughly measured by the TOF camera 62 and used to define the search range.

Precalibration Algorithm FIG. 10 is a flow diagram of a computer program that implements the precalibration software 140 shown in FIG. Referring to FIG. 10, the program starts at step 300 where variable i is set to zero. The variable i indicates the number of points selected by the user for pre-calibration of the TOF camera 62.

  The program includes step 302 following step 300. Here, it is determined whether or not the variable i is larger than the constant MAX, and the control flow branches in two directions according to the determination. The constant MAX is the number of points (pixels) used for preliminary calibration and is determined in advance. As mentioned elsewhere in the specification, MAX must be >> 6.

The program further includes the following steps. This is executed when the determination in step 302 is NO, and step 304 for detecting the coordinates (x _oi , y _oi , z _oi ) of the pixel selected by the user in the TOF camera image, and the storage area Range in the RAM 96 are performed. A step 306 for storing coordinates (x _oi , y _oi , z _oi ) in [i], a step 308 for detecting the coordinates (x _Li , y _Li ) of the pixel selected by the user in the left camera image, and a RAM 96 Step 310 for storing the coordinates (x _Li , y _Li ) in the storage area Left [i] within the storage area, and Step 312 for detecting the coordinate values (x _Ri , y _Ri ) of the pixel selected by the user in the right camera image. Step 314 for storing the coordinates (x _Ri , y _Ri ) in the storage area Right [i] in the RAM 96, and incrementing the variable i by 1 Step 316. After step 316, control flow returns to step 302.

Program further, the determination of step 302 is executed when it is YES, the step 318 of calculating the mapping matrix A _L and A _R for mapping each pixel of the TOF camera image to the left and right of the camera image, following step 318 Storing 320 the parameters of the mapping matrices A _L and A _R in the TOF mapping parameter memory 142 in the RAM 96 shown in FIG. In step 318, matrices A _L and A _R are calculated by solving Equation 4 using QR decomposition.

Disparity Search Algorithm FIG. 14 is a software flowchart for realizing the disparity search module 148 shown in FIG. Referring to FIG. 14, the program determines, for all TOF pixels, step 400 for mapping TOF pixels to pixels of the left camera image, whether the TOF pixel is in the foreground, and proceeds in two directions of the control flow. Is executed when the determination in step 401 is NO, the step 402 for performing normal background search, and the determination corresponding to the TOF pixel is executed when the determination in step 401 is YES. Step 404 of estimating the location of the camera image, and step 406 of performing a limited foreground search for searching the pixel corresponding to the pixel of the left camera image in the right camera image are repeated.

  Step 401 is a software implementation of the comparator 180 (see FIG. 4). If the intensity is greater than the threshold, the pixel is determined to belong to the foreground, otherwise it is determined to belong to the background. The

  In the normal background search, the search method described with reference to FIG. 9B is performed. In the limited background search, the search method described with reference to FIG. 9A is performed. The limited search greatly reduces the computational cost, so the algorithm can find the disparity between the left and right camera images in real time.

[Operation]
1 to 14, the above-described disparity estimation system 40 operates as follows. Referring to FIG. 1, calibration of stereo camera 60 is first performed offline. In the calibration process, a predetermined pattern plate is presented to the stereo camera 60. Calibration software 130 calculates calibration parameters. The parameters are stored in the calibration parameter memory 132.

Next, the TOF camera 6 is calibrated offline by the preliminary calibration software 140. In the preliminary calibration process, a predetermined pattern 340 is presented to the stereo camera 60 and the TOF camera 62. In the pre-calibration process, the stereo image is corrected by the correction software 134 and stored in the frame memory 136. As shown in FIG. 10, the preliminary calibration software 140 reads the TOF camera image and the corrected stereo image from the calibration frame memory 138 and the frame memory 136, respectively. Three sets of images (TOF camera image and left and right images) are shown in sequence on the 3D monitor 54, and the user repeatedly selects the corresponding pixels of these images via the GUI. By moving the pattern 340 and repeating the selection process, enough pixels (>> 6) are collected to solve equation (4). Mapping matrices A _L and A _R are obtained by solving Equation 4 using QR decomposition. The mapping matrix is stored in the TOF mapping parameter memory 142.

  When the calibration of the stereo camera 60 and the preliminary calibration of the TOF camera 62 are complete, the disparity estimation system 40 is ready to capture a 3D scene and generate a stream of disparity images in the disparity memory 150.

  In operation, the disparity estimation system 40 obtains a stream of stereo images from the stereo camera 60. The captured image is corrected by the correction software 134 using the calibration parameters stored in the calibration parameter memory 132 so that epipolar lines correspond to the rows of the image. The corrected image is stored in the frame memory 136. At the same time, the disparity estimation system 40 obtains a stream of TOF images from the TOF camera 62.

  For each pixel in each frame of the TOF camera image, the foreground / background mapping module 146 maps the pixel to the left camera image. Thereafter, module 146 examines the pixel intensity of the TOF camera image to determine whether the pixel belongs to the foreground object or the background. If the pixel is in the foreground, module 146 sends a high level F / G signal 176 to disparity search module 148. Otherwise, module 146 sends a low level F / G signal 176 to disparity search module 148.

  In response to the high level F / G signal 176, the disparity search module 148 estimates the pixels of the right camera image by mapping the pixels of interest in the TOF camera image to the right camera image. Thereafter, the disparity search module 148 determines the disparity of the left and right camera images using a foreground search algorithm in the limited search range.

  In response to the low level F / G signal 176, the disparity search module 148 locates the pixel in the right camera image having the same xy coordinate values as the left pixel, and searches for background disparity over a longer search range. Disparity is determined using an algorithm. When the search for the corresponding pixel is finished, the disparity is calculated and the disparity search module 148 signals the foreground / background mapping module 146 to select the next pixel of the TOF image. For each pixel, disparity search module 148 writes the disparity to the address of that pixel in the left camera image in disparity memory 150.

  When the disparity image is completed, the graphic output unit 152 selects the left camera image stored in the frame memory 136 and the disparity image stored in the disparity memory 150 and supplies them to the 3D monitor 54. At the same time, the graphic output unit 152 selects any combination of the TOF image, the stereo image, and the disparity image in accordance with the instruction of the controller 122, and supplies this to the monitor 74.

  Since the 3D monitor 54 has the ability to generate a stereo display based on the left camera image and the corresponding disparity image, the user can view the 3D image on the 3D monitor 54.

[Experimental setup]
Videre ^™ stereo vision hardware and SVS software are utilized for this implementation. Camera calibration and correction is performed automatically using the SVS library. SVS software can capture stereo video sequences and reconstruct stereo pairs of 3D data at 30 Hz with a total image resolution of 320 × 240. However, since the region of interest (ROI) of interest for 3D reconstruction is the user's face area in this experiment, the disparity search area was limited to the area around the face horopter as described above. Therefore, the reconstructed 3D data having different depths outside the face has an inaccurate 3D estimation as shown in FIG.

As shown in FIG. 1, the system is composed of a Swissrange ^™ time-of-flight (TOF) range camera (abbreviated as TOF camera) and a conventional CCD stereo camera made by Videre ^™ , which is a dense stereo. It enables a substantially parallel image channel that improves the disparity computation of the reconstruction algorithm. Stereo image capture, camera calibration and correction is performed using SRI ^™ software from SRI International ^™ . The system can capture a stereo video sequence and reconstruct the stereo pair of 3D data at an image resolution of 320 × 240 at 30 frames per second.

  The reconstructed 3D coordinate values are for a predefined world coordinate system. The world coordinate system (origin) in this implementation is defined to be the focal point of the left camera and is the right hand coordinate system.

  FIG. 15 is a screen capture for mapping from a TOF range camera to a stereo image, where the pixels mapped from the TOF camera image 430 are superimposed on the left and right camera images 432 and 434.

  FIG. 16 shows an image frame of the left camera 450, a disparity image 452, and an open GL (Open GL) plot 454 of the reconstructed 3D data, respectively. With range information from the TOF camera, 3D reconstruction for background objects is improved.

  The embodiment disclosed herein is merely an example, and the present invention is not limited to the above-described embodiment. The scope of the present invention is indicated by each claim in the claims after taking into account the description of the detailed description of the invention, and all modifications within the meaning and scope equivalent to the wording described therein are intended. Including.

It is a figure which shows the system component of the disparity estimation system 40 according to one embodiment of this invention. It is a hardware block diagram of the disparity calculation apparatus 50 implement | achieved by computer of embodiment of this invention. 3 is a functional block diagram of the entire disparity calculating apparatus 50. FIG. FIG. 6 is a diagram showing details of a foreground / background mapping module 146. FIG. 10 is a diagram showing details of a disparity search module 148. It is a figure which shows correction | amendment of a stereo image. FIG. 6 illustrates a corrected stereo image pair template search process for disparity estimation. It is a figure which shows how the disparity of the pixel 250L of the left camera image 240L is calculated. (A) shows a foreground disparity search, and (B) shows a background disparity search. FIG. 6 is a flow diagram showing the overall process steps for pre-calibration of the TOF camera 62. FIG. 4 shows a pattern 340 used in a pre-calibration process. FIG. 4 is a diagram showing how a user selects pixels of a TOF camera and a stereo camera in a pre-calibration process. It is a figure which shows how the block in the "left" camera image 240L and the "right" camera image 240R is divided | segmented into a subblock. It is a flowchart of a disparity search algorithm. It is a figure which shows the screen capture for the mapping from a TOF camera to a stereo image. It is a figure which shows the Open GL plot 454 of the image frame of the left camera 450, the disparity image 452, and 3D reconstruction data, respectively.

Explanation of symbols

40 disparity estimation system 50 disparity calculation device 54 3D monitor 60 stereo camera 62 TOF camera 70 computer 74 monitor 80 keyboard 82 mouse 90 CPU
94 ROM
96 RAM
98 Hard disk drive 130 Calibration software 132 Calibration parameter memory 134 Correction software 136 Frame memory 138 Calibration frame memory 140 Pre-calibration software

Claims (6)

A device for matching corresponding pixel pairs in a corrected stereo image pair, wherein the device is connected to a stereo camera and a range camera to receive the corrected stereo image pair and the range camera image, respectively. The stereo image pair includes a first image and a second image, and the apparatus maps a pixel in the range camera image to a pixel in the first image and a pixel in the second image. First mapping means for:
Comparison means for comparing a pixel value of the pixel in the range camera image with a threshold value;
In response to the comparison means, a first disparity search and a second disparity search are selectively performed depending on a comparison result by the comparison means, and pixels in the second image are obtained. Means for searching for a match in the first image for the pixel. The search means includes
First search means responsive to the comparison means determining that the pixel value of the pixel is higher than the threshold;
A second search means responsive to determining that the pixel value of the pixel is less than or equal to the threshold value;
The apparatus according to claim 1, wherein a search range of the first search means is shorter than a search range of the second search means. The first search means includes
Second mapping means for mapping the pixels of the range camera image to the second image in response to the comparison means determining that the pixel value of the pixel is higher than the threshold value When,
First search range defining means for defining a first search range extending on both sides of the pixel on the epipolar line of the second image mapped by the second mapping means;
A predetermined block of a predetermined size that includes the pixels in the first image and a predetermined block in the second image that includes each pixel within the first search range. First similarity calculating means for calculating a predetermined similarity measure with the size block;
First block selecting means for selecting the block in the second image having the highest similarity measure;
The first pixel selection means for selecting the pixel at the center of the block selected by the first block selection means as a match with the pixel of the first image. The device described. The second search means includes
Second search range defining means for defining a second search range that is longer than the first search range, wherein the second search range is mapped by the second mapping means. Extends only to one side of the pixel on the epipolar line of the image,
The second search means further includes a predetermined block of a predetermined size that includes the pixels in the first image, and a second block that includes each pixel in the second search range. Second similarity calculating means for calculating a predetermined similarity measure with a block of a predetermined size in the images of
Second block selection means for selecting the block in the second image having the highest similarity measure calculated by the second similarity means;
The second pixel selection means for selecting the pixel at the center of the block selected by the second block selection means as a match with the pixel of the first image. The device described. The first similarity calculation means includes:
Means for dividing each of the block in the first image and the block in the second image into a plurality of sub-blocks of the same shape;
Means for calculating an average pixel value of each of the blocks in the second image and the block in the first image;
Means for subtracting the average pixel value from the pixel value of each pixel of the sub-block;
Means for calculating an average pixel value of each pixel of the sub-block;
Means for calculating a sum of square errors between an average pixel value of the sub-blocks of the first image and an average pixel value of each of the sub-blocks of the second image block;
The apparatus of claim 3, wherein the sum of squared errors is the similarity measure. A computer-executable computer program connected to a stereo camera and a range camera to receive a corrected stereo image pair and a range camera image, respectively, the stereo image pair comprising a first image and a second image First mapping means for mapping a computer in the range camera image to a pixel in the first image when the computer program is executed on the computer;
Comparison means for comparing a pixel value of the pixel in the range camera image with a threshold value;
In response to the comparison means, a first disparity search and a second disparity search are selectively performed depending on a comparison result by the comparison means, and pixels in the second image are obtained. A computer program that operates as means for searching for a match with the pixel in the first image.

Images