CN114640801A - Vehicle-end panoramic view angle auxiliary driving system based on image fusion - Google Patents

Abstract

A vehicle-side panoramic view assisted driving system based on image fusion, comprising: an image acquisition module for acquiring 360° road information around a vehicle body, an embedded image processing device for real-time processing of images output by the image acquisition module, and a display The image display device of the panoramic image of the vehicle. Among them, the embedded image processing device and the other two devices are physically connected by cables; three fisheye cameras installed at different positions on the vehicle with a viewing angle of 180° are used to capture images, and video encoders and video capture are used to capture images. The card integrates multiple channels of analog video images into one channel. The embedded image processing device stitches the integrated digital video images through a fisheye image processing module and a panoramic image stitcher, stitches three digital video images from different angles into a panoramic image, and stitches the stitched panoramic image through the Web panoramic player. The image is displayed on an image display device. The invention can eliminate the blind area of vision when the large-scale special vehicle is running.

Description

A vehicle-side panoramic view assisted driving system based on image fusion

technical field

The invention relates to the field of safe driving of large-scale special vehicles, in particular to a vehicle-end panoramic viewing angle auxiliary driving system based on image fusion.

Background technique

In recent years, with the continuous advancement of my country's urbanization construction process, more and more large-scale special vehicles have appeared on urban roads, including but not limited to urban buses, cement tankers, and muck trucks. The emergence of large-scale special vehicles has greatly facilitated people's daily production and life. However, these large special-purpose vehicles often have the characteristics of long body and high body, which makes these vehicles have a wide range of blind spots. When driving on the road, especially when turning, serious traffic will occur due to the large blind spots. Accidents bring greater safety hazards to the driving of other vehicles or pedestrians on the road. At present, major cities in the country have implemented the policy of "stop right when turning right" for large-scale special vehicles, but traffic accidents caused by the blind spots of large-scale special-purpose vehicles have been happening all the time. Therefore, in order to improve the driving safety of large special vehicles and minimize the danger of other road vehicles or pedestrians walking, it is necessary to reduce or even completely eliminate the blind spots of vision that exist when large special vehicles drive on roads.

In order to achieve the above-mentioned purpose, various systems for reducing blind spots in the field of view of vehicles have been researched and developed. Among them, Wang Lujie et al. proposed an on-board system with front-mounted probes for automobile rearview mirrors (Wang Lujie; Ma Feilong; Stone Sculpture; et al. Vehicle-mounted system with front-mounted probes for automobile rearview mirrors [P]. Chinese Patent: CN112776729A, 2021-05-11 .), the vehicle body foreign body detection component is installed on the body to solve the problem of the blind spot of the car's visual field, but there are limitations such as complicated design and high cost. Ye Shengjuan et al. proposed an automotive imaging device for eliminating blind spots in the visual field of automobiles (Ye Shengjuan; Wang Haifeng; Yang Yingfei; et al. An automotive imaging device for eliminating blind spots in automotive visual fields [P]. Chinese Patent: CN213948279U, 2021-08-13.), By installing a fixed display on the vehicle and adjusting the fixed angle of the display to solve the problem of the blind spot of the car's field of vision, there is a problem that the viewing angle is fixed, the angle of the display needs to be adjusted manually, and the information of the surrounding environment cannot be fully displayed.

SUMMARY OF THE INVENTION

In order to overcome the above-mentioned problems of the prior art, the present invention provides a vehicle-side panoramic viewing angle assisted driving system based on image fusion, which aims to reduce or even completely eliminate the blind spot of the large-scale special-purpose vehicle, thereby improving the driving safety of the large-scale special-purpose vehicle. .

A vehicle-end panoramic viewing angle assisted driving system based on image fusion of the present invention includes an image acquisition device, an embedded image processing device and an image display device.

The image acquisition device uses three fisheye cameras to capture the 360° road environment around the vehicle body, and uses a video encoder to integrate the output images of the three fisheye cameras into one analog video, and the video capture card uses the analog video for the channel. Convert it into digital video and transmit it to the fisheye image processing module in the embedded image processing device through the cable for image processing;

The embedded image processing device includes a fisheye image processing module; a Web panorama player; a panorama image stitcher;

The fisheye image processing module is used to process the original fisheye image output by the video capture card. In order to capture a larger field of view, the original fisheye camera will cause serious distortion of the pixel information around the image, so it is necessary to correct the original fisheye image into a ring view by expanding the latitude and longitude to improve the effect of the final panorama stitching. This method mainly transforms the pixel coordinates in the 2D Cartesian coordinate system into a spherical Cartesian coordinate system by making a series of changes to the pixel coordinates in the fisheye image, and finally converts the coordinates in the spherical Cartesian coordinate system into Longitude and latitude coordinates. After that, the pixel points are mapped based on the longitude and latitude coordinates, so as to achieve the purpose of converting the fisheye image into a ring view. The specific operation steps are as follows:

1) After obtaining the original fisheye image, write a circular mask function with the center and radius of the three-view fisheye imaging to intercept the target image area, wherein the coordinate range of the pixels in the intercepted image area is as follows: Formula (1) shows:

x∈[0,cols-1],y∈[0,rows-1] (1)

Among them, x and y are the abscissa and ordinate of the pixel coordinates of the image after interception, cols is the horizontal width of the original fisheye image, and rows is the vertical width of the original fisheye image;

2) In order to control the final video resolution after image fusion, it is necessary to control the size of the output picture in step 1);

3) Convert the pixel coordinate point (x, y) of the intercepted image area from the 2D Cartesian coordinate system to the standard coordinate A (x _A , y _A ), and the conversion relationship is shown in formula (2):

Among them, x and y are the abscissa and ordinate of the pixel coordinates of the image after interception, cols is the horizontal width of the original fisheye image, and rows is the vertical width of the original fisheye image;

4) Convert the standard coordinates A (x _A , y _A ) into spherical three-dimensional Cartesian coordinates P (x _p , y _p , z _p ), and the conversion formulas are shown in formulas (3) and (4):

P(p,φ,θ) (3)

Among them, P is the radial distance of the line OP between the coordinates of a point on the sphere and the origin O, θ is the angle between OP and the z-axis, φ is the angle between the projection of OP on the xOy plane and the x-axis, r is the radius of the ball, F is the focal length of the fisheye camera, and the spherical coordinate system is converted into a Cartesian coordinate system according to formula (5):

x _p =psinθcosφ,y _p =psinθsinφ,z _p =pcosθ (5)

5) Convert the space coordinate system P into latitude and longitude coordinates, and the conversion relationship is shown in formula (6):

Among them, x _p , y _p , z _p are the coordinates of point P, latitude is the longitude coordinate, and longitude is the latitude coordinate;

6) Convert the latitude and longitude coordinates in step 5) to the pixel coordinates (x _o , y _o ) of the expanded image, and the mapping relationship is shown in formula (7):

Among them, x ₀ represents the abscissa of the pixel in the expanded image, and y ₀ represents the ordinate of the pixel in the expanded image;

7) After the pixel point mapping is completed, there will be black void points in the picture that are not mapped by the pixel points, and the cubic interpolation algorithm is used to fill these black areas to achieve the complete effect of the output image.

The panoramic image stitcher is used to stitch the three fisheye images processed by the fisheye image processing module for panoramic images. In order to ensure the continuity of the field of view after splicing, the three fisheye cameras in different directions need to be numbered in a fixed order, and the order will remain unchanged in subsequent operations. In the process of image processing, the SIFT algorithm is used to calculate the feature points of each image, which are regarded as the image local invariant description operator whose scale space, scaling, rotation and affine transformation remain unchanged. It is also necessary to find the matching feature points between adjacent images, and use the RANSAC method to further filter out the feature matching points, so as to calculate the homography matrix by finding the mapping relationship between the feature matching points. Finally, the perspective transformation of the image is performed according to the calculated homography matrix, and finally the perspective transformed images are spliced to realize the function of vehicle-end panoramic image splicing. The specific operation steps are as follows:

1) After obtaining the fisheye image processed by the fisheye image processing module, firstly perform fixed numbering on images of different angles in turn, and keep the consistency of subsequent image numbers;

2) Calculate the feature points of each image with the SIFT algorithm that comes with OpenCV, and use it as a locally invariant description operator of the image whose scale space, scaling, rotation and affine transformation remain unchanged;

3) Image splicing also needs to find matching feature points between adjacent images, so in the present invention, the method of calculating Euclidean distance measure is used to roughly match the fisheye images of three viewing angles, and then compare the nearest neighbor Euclidean distance and the next adjacent Euclidean distance. The SIFT matching method of distance filters the feature points of the two images, and selects the matching point when the ratio of the nearest neighbor Euclidean distance to the second neighbor Euclidean distance is less than 0.8;

4) further screen out the incorrect matching points by the RANSAC method for the rough matching points processed in step 3), thereby improving the accuracy of subsequent image processing, and then find out the mapping relationship between the feature points, thereby calculating the homography matrix;

5) Perform perspective transformation on the fisheye image processed by the fisheye image processing module with the homography matrix calculated in step 4), then stitch the perspective transformed images, and finally synthesize the video stream to realize the function of panoramic stitching.

The web panorama player is used to display the panorama images output by the panorama image stitcher on the web page. In order to reduce the delay of video display, the panorama player uses the rtc.js player plug-in to build a front-end player to play the video. In order to enable the front end to support the playback of panoramic video, the technology of three.js+video tag+rtc.js is used to realize the panoramic player. The panorama player mainly builds a spherical model through three.js, and uses the video tag as the surface rendering material of the sphere to map the sphere, so as to achieve the effect of projecting the panoramic video onto the sphere. The panorama player mainly builds a spherical model through three.js, and uses the video tag as the surface rendering material of the sphere to map the sphere, so as to achieve the effect of projecting the panoramic video onto the sphere. Install and browse on the embedded image processing device. browser, you can browse the panoramic image on the image display device;

The image display device establishes a physical connection with the embedded image processing device, and is used to display the panoramic image presented by the Web player.

Compared with the prior art, the beneficial effect of the present invention is: three fisheye cameras can be used to obtain 360° environmental information around the vehicle body, and three channels of analog video data are integrated into one channel of digital video data through a video encoder and a video capture card. Video data, input the digital video data to the embedded image processing device for further processing, which greatly saves the port resources of the embedded device, and the overall design cost is also relatively low. At the same time, the embedded device integrated with AI chip is used to improve the real-time processing capability and image output capability of the video. Combined with the self-designed panoramic player to realize the playback of panoramic video, it has greatly reduced or even completely eliminated the existence of large-scale special vehicles when driving. The problem of a relatively wide field of vision blind spot has a good assisted driving effect.

Description of drawings

Fig. 1 is the overall frame diagram of the system of the present invention;

Fig. 2 is the installation schematic diagram of the camera of the present invention;

Fig. 3 is the fisheye image processing flow chart of the present invention;

FIG. 4 is a process flow chart of the image splicing and fusion of the present invention.

Detailed ways

Below in conjunction with accompanying drawing, the example of the present invention is described in further detail:

As shown in Figure 1, a vehicle-side panoramic view assisted driving system based on image fusion consists of three parts: an image acquisition device, an embedded image processing device, and an image display device. Among them, the embedded image processing equipment and the image acquisition equipment and the image display equipment are physically connected by cables, and the image acquisition equipment mainly realizes the output of the multi-channel fisheye camera. The device integrates multiple channels of analog video into one channel of digital video by means of hardware encoding. While reducing the amount of data transmission, the digital video data is input into the embedded image processing device for image processing. After the embedded image processing device acquires the input digital image, it first processes the severely distorted original fisheye image into a ring view by the method of coordinate change, so as to obtain richer image information, and then processes the three-way image into a ring view. Fisheye images are image stitched to form a video stream. Finally, the spliced panoramic images are presented on the image display device on the Web side through the Web panoramic player, so as to greatly reduce or even completely eliminate the wide blind area of vision when large-scale special vehicles are driving, and improve the safety of large-scale special-purpose vehicles. .

As shown in FIG. 2 , the angle of view of the fisheye camera used in the present invention is 180°. In order to realize the collection of 360° environmental information around the vehicle body and a better image stitching effect, it is necessary to install three fisheye cameras in the same At different positions in height, and each fisheye camera needs to be separated by 120°. According to the installation schematic diagram in Figure 2, and adjust the installation positions of the three cameras on the large-scale special vehicle according to the final image display effect, the 360° panoramic environment image information around the vehicle body can be obtained, so as to obtain a good assisted driving effect.

As shown in Figure 3, the fisheye image processing module achieves the optimal effect of the output image by performing a series of transformations on the pixel coordinates in the fisheye image, mapping of pixel points, and image filling of void points. The main implementation steps are as follows:

1) After obtaining the original fisheye image, write a circular mask function with the center and radius of the three-view fisheye imaging to intercept the target image area, wherein the coordinate range of the pixels in the intercepted image area is as follows: Formula (1) shows:

x∈[0,cols-1],y∈[0,rows-1] (1)

Among them, x and y are the abscissa and ordinate of the pixel coordinates of the image after interception, cols is the horizontal width of the original fisheye image, and rows is the vertical width of the original fisheye image;

2) In order to control the final video resolution after image fusion, it is necessary to control the size of the output picture in step 1);

3) Convert the pixel coordinate point (x, y) of the intercepted image area from the 2D Cartesian coordinate system to the standard coordinate A (x _A , y _A ), and the conversion relationship is shown in formula (2):

Among them, x and y are the abscissa and ordinate of the pixel coordinates of the image after interception, cols is the horizontal width of the original fisheye image, and rows is the vertical width of the original fisheye image;

4) Convert the standard coordinates A (x _A , y _A ) into spherical three-dimensional Cartesian coordinates P (x _p , y _p , z _p ), and the conversion formulas are shown in formulas (3) and (4):

P(p,φ,θ) (3)

Among them, P is the radial distance of the line OP between the coordinates of a point on the sphere and the origin O, θ is the angle between OP and the z-axis, φ is the angle between the projection of OP on the xOy plane and the x-axis, r is the radius of the ball, F is the focal length of the fisheye camera, and the spherical coordinate system is converted into a Cartesian coordinate system according to formula (5):

x _p =psinθcosφ,y _p =psinθsinφ,z _p =pcosθ (5)

5) Convert the space coordinate system P into latitude and longitude coordinates, and the conversion relationship is shown in formula (6):

Among them, x _p , y _p , z _p are the coordinates of point P, latitude is the longitude coordinate, and longitude is the latitude coordinate;

6) Convert the latitude and longitude coordinates in step 5) to the pixel coordinates (x _o , y _o ) of the expanded image, and the mapping relationship is shown in formula (7):

Among them, x ₀ represents the abscissa of the pixel in the expanded image, and y ₀ represents the ordinate of the pixel in the expanded image;

7) After the pixel point mapping is completed, there will be black void points in the picture that are not mapped by the pixel points, and the cubic interpolation algorithm is used to fill these black areas to achieve the complete effect of the output image.

As shown in FIG. 4 , the technical method of the panoramic image stitcher includes the extraction and calculation of image feature points, the matching of matching feature points between adjacent images, the finding of the mapping relationship between the feature points, the calculation of the homography matrix, and the perspective of the image. Transformation and image stitching, etc., the main implementation steps are as follows:

1) After obtaining the fisheye image processed by the fisheye image processing module, firstly perform fixed numbering on images of different angles in turn, and keep the consistency of subsequent image numbers;

2) Calculate the feature points of each image with the SIFT algorithm that comes with OpenCV, and use it as a locally invariant description operator of the image whose scale space, scaling, rotation and affine transformation remain unchanged;

3) Image splicing also needs to find matching feature points between adjacent images, so in the present invention, the method of calculating Euclidean distance measure is used to roughly match the fisheye images of three viewing angles, and then compare the nearest neighbor Euclidean distance and the next adjacent Euclidean distance. The SIFT matching method of distance filters the feature points of the two images, and selects the matching point when the ratio of the nearest neighbor Euclidean distance to the second neighbor Euclidean distance is less than 0.8;

4) further screen out the incorrect matching points by the RANSAC method for the rough matching points processed in step 3), thereby improving the accuracy of subsequent image processing, and then find out the mapping relationship between the feature points, thereby calculating the homography matrix;

5) Perform perspective transformation on the fisheye image processed by the fisheye image processing module with the homography matrix calculated in step 4), then stitch the perspective transformed images, and finally synthesize the video stream to realize the function of panoramic stitching.

The content described in the embodiments of the present specification is only an enumeration of the realization forms of the inventive concept, and the protection scope of the present invention should not be regarded as limited to the specific forms stated in the embodiments, and the protection scope of the present invention also extends to those skilled in the art. Equivalent technical means that can be conceived by a person based on the inventive concept.

Claims (3)

1. The utility model provides a car end panorama visual angle driver assistance system based on image fusion which characterized in that: the method comprises the following steps: the system comprises an image acquisition module, an embedded image processing device and an image display device, wherein the image acquisition module is used for acquiring 360-degree road environment information around a vehicle body;

the image acquisition equipment adopts three fisheye cameras to acquire a 360-degree road environment around a vehicle body, integrates output images of the three fisheye cameras into one analog video by using a video encoder, converts the analog video into a digital video by using a video acquisition card, and transmits the digital video to a fisheye image processing module in the embedded image processing equipment through a cable for image processing;

the embedded image processing equipment comprises a fisheye image processing module; a Web panorama player; a panoramic image splicer;

the fisheye image processing module is used for processing the original fisheye image output by the video acquisition card; the original fisheye image is corrected into a ring view by adopting a longitude and latitude unfolding mode to improve the final panoramic stitching effect: the method comprises the following steps of transforming pixel coordinates in a 2D Cartesian coordinate system into a spherical Cartesian coordinate system by carrying out a series of changes on the pixel coordinates in a fisheye image, finally converting the coordinates in the spherical Cartesian coordinate system into longitude and latitude coordinates, and then mapping pixel points based on the longitude and latitude coordinates to convert the fisheye image into a circular view, wherein the specific operation steps are as follows:

1) after an original fisheye image is obtained, writing a circular mask function by using the circle center and the radius of three-view fisheye imaging to intercept an image area of a target, wherein the coordinate range of a pixel point of the intercepted image area is as shown in formula (1):

x∈[0,cols-1],y∈[0,rows-1] (1)

wherein x and y are respectively an abscissa and an ordinate of a pixel coordinate of the intercepted image, cols is a transverse width of the original fisheye image, and rows is a longitudinal width of the original fisheye image;

2) in order to control the resolution of the video finally subjected to image fusion, the size of the output picture in the step 1) needs to be controlled;

3) converting the pixel coordinate point (x, y) of the intercepted image area from the 2D Cartesian coordinate system to a standard coordinate A (x)_A,y_A) The conversion relationship is shown in equation (2):

wherein x and y are respectively an abscissa and an ordinate of a pixel coordinate of the intercepted image, cols is a transverse width of the original fisheye image, and rows is a longitudinal width of the original fisheye image;

4) the standard coordinate A (x)_A,y_A) Three-dimensional Cartesian coordinates P (y) converted into spherical shape_p,y_p,z_p) The conversion formula is shown in formulas (3) and (4):

P(p,φ,θ) (3)

wherein, P is the radial distance of the connecting line OP between one point coordinate on the sphere and the origin O, theta is the included angle between OP and the z axis, phi is the included angle between the projection of OP on the xOy plane and the x axis, r is the radius of the sphere, F is the focal length of the fisheye camera, and the spherical coordinate system is converted into a Cartesian coordinate system according to the formula (5):

x_p＝psinθcosφ,y_p＝psinθsinφ,z_p＝pcosθ (5)

5) converting the space coordinate system P into longitude and latitude coordinates, wherein the conversion relation is shown as a formula (6):

wherein x is_p,y_p,z_pIs the coordinate of point P, latitude is the longitude coordinate, longtude is the latitude coordinate;

6) converting and mapping the longitude and latitude coordinates in the step 5) into pixel coordinates (x) of an expansion map_o,y_o) The mapping relationship is shown in formula (7):

wherein x is₀The pixel abscissa, y, in the expanded view is shown₀Indicating the pixel ordinate in the unfolded image;

7) after the pixel point mapping is completed, black void points which are not mapped by the pixel points appear in the picture, and the cubic interpolation algorithm is utilized to fill the black areas so as to achieve the effect of outputting the complete image;

panoramic picture splicer carries out panoramic picture's concatenation with three fisheye image after fisheye image processing module handles, includes: in order to ensure the continuity of the spliced vision, the fisheye cameras in three different directions need to be numbered according to a fixed sequence, and the sequence is always kept unchanged in the subsequent operation; in the image processing process, calculating the characteristic points of each image by using an SIFT algorithm, and taking the characteristic points as image local invariant description operators for keeping scale space, scaling, rotation and affine transformation unchanged; matching feature points between adjacent images are also required to be searched, and the RANSAC method is used for further screening out the feature matching points, so that a homography matrix is calculated by searching the mapping relation between the feature matching points; finally, perspective change is carried out on the images according to the homography matrix obtained through calculation, and finally the images after perspective change are spliced to realize the function of splicing the panoramic images at the vehicle end; the specific operation steps are as follows:

(1) when the fisheye image processed by the fisheye image processing module is obtained, images at different angles are sequentially and fixedly numbered, and the consistency of the numbers of the subsequent images is kept;

(2) calculating the characteristic point of each image by using an OpenCV self-contained SIFT algorithm, and taking the characteristic point as an image local invariant description operator with unchanged scale space, scaling, rotation and affine transformation;

(3) matching feature points between adjacent images are searched for in image splicing, rough matching is carried out on fisheye images at three visual angles by adopting a method for calculating Euclidean distance measure, then screening is carried out on the feature points of the two images by using an SIFT matching mode for comparing nearest neighbor Euclidean distance with next neighbor Euclidean distance, and when the ratio of the nearest neighbor Euclidean distance to the next neighbor Euclidean distance is less than 0.8, the matching feature points are selected as the matching points;

(4) further screening out mismatching points from the rough matching points processed in the step (3) by using a RANSAC method, thereby improving the precision of subsequent image processing, and then finding out the mapping relation among the characteristic points, thereby calculating a homography matrix;

(5) carrying out perspective transformation on the fisheye image processed by the fisheye image processing module by using the homography matrix obtained by the calculation in the step (4), then splicing the images subjected to the perspective transformation, and finally synthesizing a video stream to realize the function of panoramic splicing;

the Web panoramic player displays the panoramic image output by the panoramic image splicer on a webpage; in order to reduce the time delay of video display, the Web panoramic player adopts rtc.js player plug-in to construct a front-end player to play the video; in order to enable the front end to support the playing of the panoramic video, the panoramic playing is realized by adopting the technology of three.js + video tag + rtc.js; js, the Web panoramic player establishes a spherical model, and maps a sphere by taking a video label as a sphere surface rendering material, so as to achieve the effect of projecting a panoramic video onto the sphere; the Web panoramic player establishes a spherical model through three.js, maps a sphere by taking a video label as a rendering material on the surface of the sphere, and projects a panoramic video onto the sphere; installing a browser on the embedded image processing equipment, and browsing the panoramic image on the image display equipment;

and the image display equipment and the embedded image processing equipment are in physical connection, and display the panoramic image presented by the Web panoramic player.

2. The vehicle-end panoramic view auxiliary driving system based on image fusion as claimed in claim 1, characterized in that: the embedded image processing apparatus selects Atlas 200 acceleration module as the AI computation chip.

3. The vehicle-end panoramic view auxiliary driving system based on image fusion is characterized in that: the viewing angle of the fisheye camera is 180 °, three fisheye cameras are mounted at different positions at the same height, and 120 ° is required between each fisheye camera.

Images