CN115964446A - A method for interactive processing of radar data based on mobile terminal - Google Patents

Abstract

A radar data interaction processing method based on a mobile terminal belongs to the field of automatic driving. According to the invention, the original point cloud data is quickly and accurately processed at the calculation generation end, and the radar data is sent to the mobile end for display through the calculation generation end, so that the problems of space occupation and difficulty in movement of display equipment are solved, the method has the advantage of high flexibility, and more testers can participate in debugging work at the same time; in addition, the radar data are processed at the mobile terminal, barrier information is represented through a simple model, more passengers can understand the radar data, and each decision made by the unmanned vehicle can be fully trusted when the unmanned vehicle is taken. The invention solves the problems that a calculation generation end is difficult to move and an ordinary user does not understand radar data.

Description

A method for interactive processing of radar data based on mobile terminal

technical field

The invention belongs to the field of automatic driving.

Background technique

Self-driving vehicles make correct decisions and controls through perceived data, and perception is inseparable from sensors, such as radars and cameras. Among them, the radar emits a detection signal (laser beam) to the target, and then compares the received signal (target echo) reflected from the target with the transmitted signal, and after proper processing, the relevant information of the target can be obtained, such as the target distance. , azimuth, height, speed, attitude, and even shape parameters, so as to detect, track and identify the target. However, the raw data of radar is complex point cloud data. For workers who have been engaged in the field of autonomous driving, it is not too difficult to understand these data. Or, an easy-to-understand radar data presentation is very important. Therefore, a radar data interactive processing method based on the mobile terminal is proposed. The calculation generation terminal processes the original radar data, transmits the radar data to the mobile terminal through the UDP protocol, and uses the corresponding algorithm to display obstacles, which is easy to understand, flexible and convenient. Interactive display The advantages.

In traditional unmanned vehicles, the raw data is usually displayed on the central control screen. To understand the data perceived by the vehicle and the correctness of the decisions made in real time, testers must be in the vehicle. As far as the display interface is concerned, the "Display Screen Panel with Graphical User Interface for Lidar Data" (public number: CN307549670S) has designed a Lidar display interface, which is suitable for computers, mobile phones, and tablet devices, but only for Lidar data. Playback display, and in unmanned driving, the vehicle runs fast and needs to be decision-making and controlled by real-time perceived data. This method cannot meet the real-time radar data processing and display requirements; as far as radar data processing is concerned, the ground Point cloud filtering method, system, equipment and storage medium" (public number: CN115166700A) through the structured coding of the point cloud, using the conditional query of neighbor points to retain the environment points, and filter out the ground points and noise points in the point cloud , this method can guarantee accuracy by filtering out ground points through information such as neighbor points, but cannot guarantee real-time performance.

In view of the above shortcomings, the present invention proposes a method for interactive processing of radar data based on the mobile terminal, which quickly and accurately processes the original point cloud data at the calculation generation terminal, and sends the radar data to the mobile terminal for display through the calculation generation terminal processing, solving the problem of The problem of display equipment occupying space and being difficult to move has the advantages of high flexibility, allowing more testers to participate in the debugging work at the same time; in addition, the radar data is processed on the mobile terminal and represented by a simple model Obstacle information allows more passengers to understand the radar data, and can fully trust every decision made by the unmanned vehicle when riding in the unmanned vehicle. The purpose of the present invention is to provide a method for interactive processing of radar data based on a mobile terminal, so as to solve the problems that the computing generation terminal is difficult to move and ordinary users do not understand radar data.

Contents of the invention

The whole system includes data acquisition unit, processing unit and display unit. Among them, the data acquisition unit and the processing unit are completed on the calculation generation end, and the display unit is completed on the mobile end. The data acquisition unit includes two parts of data, one part obtains a set of unordered original point cloud data through the vehicle-mounted 3D lidar, and the other part obtains vehicle information through the CAN bus; the processing unit clusters the point cloud data into multiple independent subsets , and on this basis, carry out target classification and recognition; the display unit displays vehicle information and recognized obstacles on the mobile terminal.

1. Acquisition of point cloud data

Obtain a set of unordered original point cloud data through the vehicle-mounted 3D lidar, where the point cloud data should at least include the point cloud points of the road area scanned by the vehicle during driving, and each point cloud point data should include coordinate information, and With time stamp and direction of the beam.

2. Processing of point cloud data

The processing unit processes the original point cloud data. The preprocessing unit includes ground point cloud data filtering and target clustering and segmentation operations. The recognition unit calculates the geometric data of the clustered point cloud to identify obstacles.

1) Convert the raw point cloud data into a depth image, each pixel of this depth image stores the measured distance from the sensor to the object.

2) Filter out the ground point cloud data

(1) traverse all point cloud points in the point cloud collection in the point cloud graph, and obtain the road surface height of the driving road corresponding to each point cloud point;

(2) Based on the point cloud coordinate information of each point cloud point in the point cloud map, determine the height coordinates of each point cloud point relative to the driving road;

(3) If the height coordinate of any point cloud point is less than or equal to the corresponding road height, the point cloud point is removed from the point cloud collection to reduce the amount of point cloud data.

3) Clustering point cloud data

(1) Point cloud data clustering

Through the nearest neighbor query algorithm of KD-Tree, calculate the Euclidean distance from the neighbor point to the target point, and perform clustering according to the distance. Repeat the calculation process until all new points have been calculated.

(2) Obstacle recognition

For each cluster, a rectangle with the minimum area around the obstacle is obtained based on the minimum convex hull method, and a cubic frame is obtained. Feature extraction and classification are performed on the cube frame area to identify the target obstacle. Obstacle information includes corner coordinates, type and id.

3. Acquisition of vehicle status information

The vehicle information is obtained through the CAN bus, and the vehicle status information includes steering wheel angle, power, speed, gear position, accelerator opening and brake opening.

4. Data communication

1) UDP protocol to transmit data

(1) Carry out frame selection based on the obstacle information identified by the radar point cloud data. In order to solve the problem of huge amount of point cloud data, the corner coordinates and obstacle categories of the obstacle rectangle are transmitted during data transmission.

(2) Transfer byte data through UDP protocol

It is stipulated that every eight bytes is an obstacle information or vehicle information. The transmission data occupies two ports, and the data transmitted by each port is 1498-bit byte data. One port sends vehicle status information, the other port sends radar data information, and the two ports send cyclically.

5. Data display interaction

1) The mobile terminal receives obstacle data and vehicle data.

2) Coordinate system conversion for obstacle information

The radar obstacle data takes the center point of the rear of the vehicle as the coordinate origin, and the unit is meters, thus forming a vehicle coordinate system; when displayed on the mobile terminal, an image coordinate system is formed, and controls are drawn in the middle of the interface to display lane information, to display The upper left corner of the lane is the coordinate origin, and the unit is pixel. It is necessary to convert the vehicle coordinate system to the image coordinate system. In the vehicle coordinate system, take the origin of the coordinates to the right as the X axis, and the upwards as the Y axis; in the image coordinate system, take the origin of the coordinates as the x axis to the right, and the downwards as the y axis, then the coordinate conversion formula is

x'=w/2+X'*(w/m)

y'=h-Y'*(h/n)

Where w is the pixel width of the lane displayed on the screen, h is the pixel height of the lane displayed on the screen, m is the maximum lateral distance (unit: meter) between the displayed obstacle and the vehicle, and n is the distance between the displayed obstacle and the vehicle The maximum longitudinal distance (unit: meter), X' is the actual horizontal distance from the obstacle to the vehicle (unit: meter), Y' is the actual longitudinal distance from the obstacle to the vehicle (unit: meter), and x' is the calculated The pixel horizontal distance (unit: pixel) between the obstacle in the interface and the vehicle, and y' is the calculated pixel vertical distance (unit: pixel) between the obstacle and the vehicle in the interface;

3) Develop the mobile terminal interface to display vehicle information (vehicle speed, battery, steering wheel angle, accelerator opening, brake opening, gear position) and obstacle information.

Description of drawings

Fig. 1 is the work flowchart of the radar data interaction processing method based on the mobile terminal of the present invention;

FIG. 2 is a schematic diagram of coordinate conversion based on the mobile terminal-based radar data interactive processing method of the present invention;

Fig. 3 is the mobile terminal interface of the radar data interactive processing method based on the mobile terminal in the present invention;

Detailed ways

Referring to the accompanying drawings, through the description of the embodiments, the specific implementation of the present invention, such as the shape, structure, mutual position and connection relationship between the various parts, the function and working principle of each part, and the manufacturing process And the method of operation and use, etc., are described in further detail to help those skilled in the art have a more complete, accurate and in-depth understanding of the inventive concept and technical solution of the present invention.

In this embodiment, the calculation generator uses Jetson AGX Orin, which is the smallest, most powerful, and most energy-efficient AI supercomputer released by NVIDIA, capable of performing 200 trillion operations per second (TOPS). The mobile terminal is Huawei M6 tablet, which is small in size and can basically meet the display requirements.

Example 1 Figure 1 is the working flow chart of the radar data interactive processing method based on the mobile terminal of the present invention. As shown in the figure, firstly, the original point cloud data is obtained through the vehicle-mounted three-dimensional laser radar, and the vehicle information is obtained through the CAN bus, including steering wheel angle, power, Information such as speed, gear position, accelerator opening and brake opening; then filter the ground point cloud data to reduce the amount of point cloud data, and use a clustering algorithm to filter the point cloud data for the same obstacle Carry out clustering and identify obstacles through the data after clustering; use UDP-based data transmission protocol to transmit vehicle information and point cloud data after processing and identification to the mobile terminal for display in real time.

Most lidars provide raw data in the form of individual ranging readings for each laser beam, with time stamps and the orientation of the beams, allowing the data to be converted directly into a depth image. Each pixel of this depth image stores the measured distance from the sensor to the object. When filtering ground point cloud data, traverse all point cloud points in the point cloud set in the point cloud map, and query the road surface height of the driving road corresponding to each point cloud point; based on each point cloud point in the point cloud map Point cloud coordinate information, determine the height coordinates of each point cloud point relative to the driving road; if the height coordinate of any point cloud point is less than or equal to the corresponding road surface height, remove the point cloud point from the point cloud collection, thus Complete the filtering of ground point cloud data and reduce the amount of point cloud data. Afterwards, the point cloud data is clustered in a European way, and the Euclidean distance from the neighbor point to the target point is calculated through the KD-Tree nearest neighbor query algorithm, and clustered according to the distance. Repeat the calculation process until all new points have been calculated. After obtaining the point cloud of each obstacle, the points around the obstacle are obtained based on the minimum convex hull method. Based on these points, the rectangle surrounding the minimum area is obtained. When surrounding the 3D object, a cubic frame will be obtained . Then by calculating the geometric relationship of the clusters in each cube box, the target obstacle is identified based on the calculation results.

Embodiment 2 UDP protocol data transmission process

The data is transmitted through the UDP protocol, and the data transmitted by the protocol occupies two ports, and the data transmitted by each port is 1498-bit byte data. One port sends vehicle status information, the other port sends radar data information, and the two ports send cyclically. In the two ports, the 1st to 6th bytes describe the source device MAC address, the 7th to 14th bytes describe the target address MAC address, the 15th to 24th bytes describe the source device IP address, and the 25th to 34th bytes describe the target device IP address, the 35th to 38th bytes describe the data transmission port of the source device, the 39th to 42nd bytes describe the data receiving port of the target device, and the following bytes are data bytes. Each obstacle data transmitted by the calculation generation end at the first port occupies 72 bytes, the first 16*4 bytes represent a corner point coordinate information, each 8 bytes represent x or y coordinates, and the last 8 bytes Represents type information. In order to transmit more obstacle data, only four corner coordinates are transmitted, such as left front upper, right front upper, left rear lower, right rear lower, and the rectangular frame of the obstacle can be restored according to the four corner coordinates. For the vehicle information transmitted by the calculation generator at the second port, every 8 bytes represent a piece of data, for example, the 43rd to 50th bytes represent the steering wheel angle information. Since the transmitted vehicle information data is limited, the remaining unoccupied bytes can be used to transmit obstacle data or add other vehicle information later.

Embodiment 3 FIG. 2 is a schematic diagram of the coordinate transformation of the radar data interactive processing method based on the mobile terminal in the present invention. The pixel origin (0,0) of the image coordinate system displays the upper left corner of the lane on the screen, the right is the x axis, and the downward is the y axis. Assume that the pixel width of the lane displayed on the mobile terminal is w, and the height is h, that is, the pixel coordinates of the lower right corner of the lane are (w, h). Taking Huawei Tablet M6 as an example, the lane pixel width displayed on the interface is w=1800, and the pixel height h= 1600. The origin of the actual vehicle coordinates is the center point of the rear of the vehicle, the X axis is to the right, and the Y axis is to the front. For example, there are 3 lanes, each lane is 4 meters wide and 50 meters long, and the vehicle is located directly below the middle lane, then it will display The obstacle distance from the vehicle is the horizontal distance range X∈[-6,6], and the longitudinal distance range Y∈[0,50]. Therefore, the coordinate conversion formula is as follows:

x'=1800/2+X'*(1800/12)

y’=1600–Y’*(1600/50)

For example, the coordinates of the obstacle received by the vehicle are (2,3), that is, the actual lateral distance from the vehicle is X=2m, and the longitudinal distance Y=3m, then the position displayed in the interface lane is

x'=1800/2+2*(1800/12)=1200px

y'=1600–3*(1600/50)=1504px

Embodiment 4 Fig. 3 is the mobile terminal interface of the radar data interactive processing method based on the mobile terminal of the present invention, three lanes are displayed in the interface, each lane is 4 meters wide and 50 meters long, and the vehicle is located directly below the middle lane, for the sake of beautiful display , the plane of the lane is rotated 30 degrees toward the interface. The upper part of the interface displays battery power, vehicle speed, and steering wheel angle in turn from left to right, and they are all displayed in the form of integers. The accelerator opening and brake opening are displayed on the left, and the accelerator and brake are displayed by a progress bar, and the unit is a percentage. For example, the accelerator is 34%, and 34% of the progress bar is blue. The gear position information of the right vehicle. For obstacles, obstacles within ±6 meters left and right of the vehicle and 50 meters in front of the vehicle are displayed on the interface. According to the obstacle type data sent by the calculation generation end, the corresponding obstacle model is displayed, and the obstacle model is translated by the association between the transmission data frame and the obstacle id between the frames to ensure the smoothness of the obstacle movement.

Claims (1)

1. A method for interactive processing of radar data based on a mobile terminal, characterized in that: 1. Acquisition of point cloud data Obtain a set of unordered original point cloud data through the vehicle-mounted 3D lidar, where the point cloud data should at least include the point cloud points of the road area scanned by the vehicle during driving, and each point cloud point data should include coordinate information, and with time stamp and direction of the beam; 2. Point cloud data processing The processing unit processes the original point cloud data, where the preprocessing unit includes ground point cloud data filtering and target clustering and segmentation operations, and the recognition unit calculates the geometric data of the clustered point cloud to identify obstacles; 1) Convert the raw point cloud data into a depth image, each pixel of this depth image stores the measured distance from the sensor to the object; 2) Filter out the ground point cloud data (1) traverse all point cloud points in the point cloud collection in the point cloud graph, and obtain the road surface height of the driving road corresponding to each point cloud point; (2) Based on the point cloud coordinate information of each point cloud point in the point cloud map, determine the height coordinates of each point cloud point relative to the driving road; (3) If the height coordinate of any point cloud point is less than or equal to the corresponding road height, remove the point cloud point from the point cloud collection to reduce the amount of point cloud data; 3) Clustering point cloud data (1) Point cloud data clustering Through the nearest neighbor query algorithm of KD-Tree, calculate the Euclidean distance from the neighbor point to the target point, and perform clustering according to the distance. Repeat the calculation process until all new points are calculated; (2) Obstacle recognition For each cluster, the rectangle with the smallest area around the obstacle is obtained based on the minimum convex hull method, and a cube frame is obtained; feature extraction and classification are performed on the cube frame area to identify the target obstacle; obstacle information includes corner coordinates , type and id; 3. Acquisition of vehicle status information Obtain vehicle information through the can bus, vehicle status information includes steering wheel angle, battery, speed, gear position, accelerator opening and brake opening; 4. Data communication 1) UDP protocol to transmit data (1) Frame selection of the obstacle information identified by the radar point cloud data; during data transmission, the corner coordinates and obstacle category of the obstacle rectangular frame are transmitted; (2) Transfer byte data through UDP protocol It is stipulated that every eight bytes is an obstacle information or vehicle information; the transmission data occupies two ports, and the data transmitted by each port is 1498-bit byte data; one port sends vehicle status information, and the other port sends radar data information , the two ports send cyclically; 5. Data display interaction 1) The mobile terminal receives obstacle data and vehicle data; 2) Coordinate system conversion for obstacle information The radar obstacle data takes the center point of the rear of the vehicle as the coordinate origin, and the unit is meters, thus forming a vehicle coordinate system; when displayed on the mobile terminal, an image coordinate system is formed, and controls are drawn in the middle of the interface to display lane information, to display The upper left corner of the lane is the coordinate origin, and the unit is pixel. It is necessary to convert the vehicle coordinate system to the image coordinate system; The origin of the coordinates is the x-axis to the right, and the y-axis is downwards, then the coordinate conversion formula is x'=w/2+X'*(w/m) y'=h-Y'*(h/n) Where w is the pixel width of the lane displayed on the screen, h is the pixel height of the lane displayed on the screen, m is the maximum lateral distance between the displayed obstacle and the vehicle, n is the maximum vertical distance between the displayed obstacle and the vehicle, X ' is the actual horizontal distance from the obstacle to the vehicle, Y' is the actual longitudinal distance from the obstacle to the vehicle, x' is the calculated pixel horizontal distance from the obstacle to the vehicle in the interface, and y' is the calculated interface The pixel longitudinal distance between the middle obstacle and the vehicle; 3) Develop the mobile terminal interface to display vehicle information and obstacle information.

Images