CN108093188B - A method for large-field video panorama stitching based on hybrid projection transformation model - Google Patents

Abstract

The present invention provides a kind of joining method of big visual field video panorama based on hybrid projection transformation model, including five big steps, step 1: down-sampled, reduction operand is carried out to video；Step 2: the detection of Sift characteristic point is extracted；Step 3: to feature points clustering, region segmentation is carried out to image using SVM method, and carry out the matching of characteristic point；Step 4: hybrid projection matrix is calculated；Step 5: interative computation, realize the registration of subsequent image sequence with merge, the final splicing for realizing panorama sketch.Time-consuming problem in big, more projection array methods that present invention effectively avoids errors in traditional single projection array method has wide application background.

Description

A method for large-field video panorama stitching based on hybrid projection transformation model

technical field

The invention relates to a method for splicing a large field of view video panorama based on a hybrid projection transformation model. The method can realize accurate and fast splicing of video image sequences, thereby generating a panorama, which can be applied to technologies such as virtual reality and belongs to computer vision. field.

Background technique

The large-field video panorama stitching technology is a hot research problem in the current video processing field. It can stitch the image sequence of a video to form a wide-angle image containing a complete scene, which solves the problems existing due to the limitation of camera equipment. The problem is that the shooting angle is small and the panoramic observation cannot be performed. The technology has been widely used in remote sensing, seabed detection and virtual reality and other fields. In the field of remote sensing, the acquisition of high-resolution images requires splicing to build a large-area panoramic image; in seabed detection, it is also necessary to stitch the acquired small-lens video information to form a panoramic image for observation and processing; and virtual reality requires more Provide users with a more intuitive and three-dimensional experience through the establishment of panoramic images.

In recent years, although panorama stitching technology has been widely studied, and there are many mature methods, there are still many challenging problems to be solved. The biggest problem is that in most of the current panorama stitching, there is an assumption: either the image and the camera are far away, and the scenes in the image can be considered to be in the same plane; or the image is shot by the camera rotating along the projection center. owned. In this way, during image registration, a homography matrix can be used as a projection transformation model for image registration.

However, this method is not robust, because it has strong randomness and freedom in actual video shooting, and basically cannot meet the above assumptions; therefore, if a homography is still used for stitching, the obtained panorama The picture is wrong. At present, most algorithms use image fusion technology in post-processing to compensate for the errors in the registration process, but this cannot fundamentally solve the problem.

Based on this, the use of multi-homography arrays for video panorama stitching has gradually begun to be studied. Among them, a typical one is the Dual-Homography Warping (DHW) method published on CVPR in 2011, which realizes the registration and stitching of the entire image by synthesizing the homography matrix of two planes. However, in the weight calculation, since the Euclidean distance needs to be calculated, and with the increase of feature points, the number of calculations will also increase to a large extent, so it will take a lot of time.

SUMMARY OF THE INVENTION

The technical solution of the present invention is to overcome the deficiencies of the prior art and provide a method for splicing a large field of view video panorama based on a hybrid projection transformation model, which consumes less time and has higher accuracy. Without any constraints, it can have a wide range of applications.

Technical solution of the present invention: The present invention aims to realize the splicing of video panorama images with a large field of view. The input is a video acquired by the camera, and the output is a panoramic image. The present invention assumes that the processed video image can be roughly divided into two planes (Background plane and foreground plane). The following introduces the technical solution for realizing large-field video panorama stitching in the present invention:

(1) Downsampling the video image sequence to ensure that there is at least 1/3 of the overlapping area between every two adjacent images;

(2) The detection of Sift feature points is performed on the overlapping area of the first two images obtained by sampling, the obtained feature point set, and feature vector extraction is performed on the feature points;

(3) Clustering the feature point set, using the support vector machine SVM method to divide the image into three regions, denoted as regions S _b , _Su and S _f , and matching the feature points;

(4) The Random Sample Consensus (RANSAC) method is used for the feature points in S _b and S _f respectively to eliminate the mismatched points, and the homography matrix is calculated, which are denoted as H _b and H _f respectively;

(5) The first two images obtained by sampling are registered. During the registration process, the pixels in the three regions S _b , S _u and S _f are processed by the hybrid projection transformation model from the second image to the first image. registration;

(6) Realize the registration of the third image to the second image through the iteration of the homography matrix, repeat step (6), and then realize the registration of subsequent image sequences, and finally perform image fusion to complete the stitching of the entire panorama Task.

The beneficial effects of the present invention compared with the prior art are:

(1) After using the point clustering method, the present invention adopts the SVM method to realize the division of different areas of the image, so as to prepare for the subsequent image registration process;

(2) In the process of image registration, an innovative hybrid projection transformation model is used, which can achieve a better stitching effect than a single projection matrix;

(3) In the process of image registration, an innovative hybrid projection transformation model is used, which is not only closer to the real situation than the traditional DHW and other hybrid projection transformation models, but also can greatly reduce the registration time and make the entire stitching process more efficient.

Description of drawings

Fig. 1 is the schematic diagram that the SVM method divides the image into three regions;

Fig. 2 method flow chart of the present invention;

Figure 3 is a graph of the experimental results of the present invention.

Detailed ways

The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

The present invention obtains video sequences by shooting with a traditional single-lens reflex camera, and realizes the stitching of video panoramas under Matlab R2016b. The flow chart of the present invention is shown in Figure 2. In order to better understand the technical scheme of the present invention, the specific embodiments of the present invention are further described below:

Step 1: Read in the video sequence, and use Matlab to downsample the acquired video sequence to reduce the computational complexity of image registration. When sampling, it is only necessary to ensure that there is about 1/3 of the overlapping area in the adjacent sampled images.

Step 2: The present invention uses the Sift operator to detect the feature points of the two adjacent images I ₁ and I ₂ and calculate the Sift feature vector.

Step 3: The two-dimensional image plane is regarded as a two-dimensional feature space, and the u and v values of the image pixels represent different eigenvalues respectively, and the K-means method is used to cluster the obtained feature point set, where K=2; then , the present invention adopts the support vector machine (SVM) method to realize the division of the image plane, as shown in FIG. 1 , wherein each small circle represents a Sift feature point. The equations of the two straight lines tangent to the support vector can be written as:

w ^T x+b ₁ =0 (1)

w ^T x+b ₂ =0 (2)

where x=(u,v) ^T .

Next, the pixel point region satisfying w ^T x+b ₁ <0 is denoted as S _b ; the pixel point region satisfying w ^T x+b ₂ >0 is denoted as S _f ; the pixel point region satisfying w ^T x+b ₁ >0 is denoted as S f ; And the pixel area where w ^T x+b ₂ <0 is denoted as _Sn .

Match the Sift feature points in the regions S _b and S _f . During the matching process, in order to improve the matching accuracy of the feature points, define any point to be matched in the region S _b (S _f ) in I ₁ (denoted as p ₁ ) The distance from the closest feature point in the feature point set in the area S _b (S _f ) in I ₂ is d ₁ , and the distance from the next closest feature point in the feature point set in the area S _b (S _f ) in I ₂ is d ₂ , then the distance ratio is recorded as:

If Ratio>ε (ε is an empirical threshold, the present invention determines after a lot of trial and error: 5≦ε), it is considered that point p ₁ has found a matching point.

By traversing all the Sift feature points in I ₁ , an initial set of matching point pairs is obtained, denoted as P _b and P _f respectively.

Step 4: Use the RANSAC method to calculate the homography matrix H for the regions S _b and S _f respectively. For two two-dimensional images, the homography transformation (or projective transformation) is satisfied between the corresponding points, as shown in (4), where (x, y, 1) ^T and (x', y', 1) ^T are the space The projected homogeneous coordinates of point X on the two images. For the region S _b , 4 sets of matching point pairs are randomly selected from the matching point pair set P _b each time to calculate H _b , and the ratio of the number of points satisfying (4) in P _b is counted, denoted as Inlier_Ratio. Finally, the homography matrix that maximizes Inlier_Ratio is selected as the final H _b .

in,

Similarly, the homography matrix corresponding to the S _f region can be calculated as H _f

Step 5: After two homography matrices are obtained, the images can be registered. For the three different regions, the present invention employs three registration models. For the pixels in the area S _b , it can be considered that they all belong to the plane where S _b is located, then these pixels are registered using the homography matrix H _b , as shown in (6), where (x, y, 1) ^T is the homogeneous coordinate of a pixel in I ₂ , and (x', y', 1) ^T is the point that is transformed to the corresponding point coordinate in I ₁ by projection.

For the pixel points in the area S _f , similarly, the transformation shown in (7) can be used to perform projective transformation on the points.

For the pixels in Su, the present invention calls them " _Undefined pixels", that is, it is impossible to accurately determine which plane these points belong to. In view of this problem, the present invention proposes the following methods to solve it.

(1) For all points in Su, define the _homography matrix:

H _ij = w _b H _b +w _f H _f (8)

where H _ij represents the homography matrix corresponding to the pixel at (i, j), H _b and H _f represent the homography matrix corresponding to the two regions of the image respectively, w _b and w _f are the homography corresponding to the two planes, respectively Weight coefficients for the matrix.

(2) Calculate the distance from the pixel point in each _{Su to the nearest feature point in the feature point set of the regions S b and} S _f , which are denoted as _{db and d f respectively} . The present invention proposes to use the Manhattan distance in the process of distance calculation. Instead of the traditional Euclidean distance, this can save time to a great extent. The weight calculation formula is as follows:

w _b =d _f /(d _b +d _f ) (9)

w _f =d _b /(d _b +d _f ) (10)

In summary, the mixed projection transformation model of each pixel can be obtained:

Among them, background represents the background area, undefined area represents the ambiguous area, and foreground represents the foreground area.

After the above steps, the registration of the two images is achieved.

Step 6: After the above five steps, the registration work of the two image sequences is realized, and then, the registration and splicing are realized for the third image I ₃ and the subsequent image sequences. The present invention adopts the homography matrix iteration method to achieve this purpose. For the convenience of clear description, the subscript of H _ij is omitted in this step, and it is denoted as H,

(1) For the region where I ₂ and I ₃ overlap, Equation (12) can be used to directly perform iterative projection transformation,

H ^3→1 =H ^2→1 (H ^3→2 (I ₃ )) (12)

(2) For the non-overlapping regions of I ₂ and I ₃ in I ₃ , the weighted iterative projection transformation can be performed using Eq. (13)

Wherein, the present invention defines K as the Sift feature point set in the overlapping area of I ₂ and I ₃ , and defines the weight λ _q as λ _q =1/d_Manh, where d_Manh represents the relationship between H ^3→2 (p) and q Manhattan distance. Then normalize λ _q so that ∑λ _q =1,

In summary:

After the above steps, the projection matrix of the third image relative to the first image can be calculated, and then the projection transformation is performed, thus realizing the registration of the third image. This method can be used iteratively for subsequent image sequences.

Finally, the registered images are fused. The present invention adopts the traditional fade-in and fade-out fusion method (ie, linear interpolation fusion algorithm), and uses a linear weighting function to achieve smooth transition processing near the boundary of the overlapping area. The weighting function is as follows The formula shows:

where I ₁ and I ₂ represent two adjacent images,

The validity and accuracy of the present invention have been verified through experiments, and good splicing results have been obtained. The experimental results are shown in Figure 3, where (a) represents the four images in the video sequence, and (b) represents the panorama calculated from these four images. The biggest advantage of the present invention is that the SVM method is used to divide the image into regions, and based on this, the projection transformation (homography transformation) of the image is performed in different regions, which effectively avoids the large error and multiple projections in the traditional single projection array method. The time-consuming problem in the matrix method has a good prospect in the technical fields such as virtual reality.

Claims (3)

1. a method for stitching a large field of view video panorama based on a hybrid projection transformation model, is characterized in that: comprise the following steps: (1) Downsampling the video image sequence to ensure that there is at least 1/3 of the overlapping area between every two adjacent images; (2) The detection of Sift feature points is performed on the overlapping area of the first two images obtained by sampling, the obtained feature point set, and feature vector extraction is performed on the feature points; (3) Cluster the feature point set, use the support vector machine SVM method to divide the image into three regions, denoted as regions S _b , S _u and S _f , and perform the corresponding regions S in I ₁ and I ₂ The matching of _b and S _f feature points; (4) The Random Sample Consensus (RANSAC) method is used for the feature points in S _b and S _f respectively to remove the mismatched points, and the relationship between the two-dimensional coordinates between the two images is calculated, that is, the homography matrix, which is recorded as H _b , H _f ; (5) The first two images obtained by sampling are registered. During the registration process, the pixels in the three regions S _b , S _u and S _f are processed by the hybrid projection transformation model from the second image to the first image. registration; (6) Realize the registration of the third image to the first image through the iteration of the homography matrix, repeat step (6), and then realize the registration of subsequent image sequences, and finally perform image fusion to complete the stitching of the entire panorama Task; In the step (5), the hybrid projection transformation model is implemented as follows: (51) For the pixels in the area S _b , use the homography matrix H _b during registration, where I ₁ and I ₂ represent the first and second images, respectively, (x, y, 1) ^T is I ₂ The homogeneous coordinates of a pixel in I 1 , (x', y', 1) ^T is the corresponding point coordinate of the point transformed to I ₁ by projection, (52) For the pixel points in the area S _f , use the transformation shown in (4) to perform projection transformation on the points; (53) For the pixels in Su, define the _homography matrix: H _ij = w _b H _b +w _f H _f (5) Wherein H _ij represents the homography matrix corresponding to the pixel at (i, j), H _b and H _f represent the homography matrix corresponding to the two image regions S _b and S _f respectively, w _b and w _f are two The weight coefficients of the image regions S _b and S _f corresponding to the homography matrix; Calculate the Manhattan distance from each pixel in S _u to the nearest feature point in the feature point set of regions S _b and S _f , denoted as _{db and d f respectively} , and the weight calculation formula is as follows: w _b =d _f /(d _b +d _f ) (6) w _f =d _b /(d _b +d _f ) (7) The hybrid projection model is expressed as: Wherein, foreground is a foreground area, namely S _f , undefined area is an ambiguous area, namely Su, and background is _{a background area, namely S b .} 2. a kind of method based on the large field of view video panorama stitching of hybrid projection transformation model according to claim 1, is characterized in that: in step (3), when image is carried out area division, (1) After using the K-means method for feature point clustering, the SVM method is used to divide the image, and the two straight line equations that are tangent to the support vector are: w ^T x+b ₁ =0 (1) w ^T x+b ₂ =0 (2) Among them, x=(u,v) ^T , w, b ₁ and b ₂ represent the coefficients of the straight line equation, u represents the abscissa of the pixel, and v represents the ordinate of the pixel; (2) Denote the pixel area that satisfies w ^T x+b ₁ <0 as S _b ; Denote the pixel area that satisfies w ^T x+b ₂ >0 as S _f ; The pixel area that satisfies w ^T x+b ₁ >0 and w ^T x+b ₂ <0 is denoted as _Su to realize the division of the image area. 3. a kind of method based on the large field of view video panorama stitching of hybrid projection transformation model according to claim 1, is characterized in that: in described step (6), the registration of the 3rd and subsequent image sequence is as follows : In the non-overlapping regions of I ₂ and I ₃ , the method of weighted homography matrix iteration is used to achieve image registration, as follows: Among them, K is defined as the set of Sift feature points in the overlapping area of I ₂ and I ₃ , and the weight λ _q is defined as λ _q =1/d_Manh, where d_Manh represents the distance between H ^3→2 (p) and the Sift feature point q The Manhattan distance of , and then normalize λ _q to make Σλ _q =1.