CN113095154A - Three-dimensional target detection system and method based on millimeter wave radar and monocular camera - Google Patents
Three-dimensional target detection system and method based on millimeter wave radar and monocular camera Download PDFInfo
- Publication number
- CN113095154A CN113095154A CN202110299442.0A CN202110299442A CN113095154A CN 113095154 A CN113095154 A CN 113095154A CN 202110299442 A CN202110299442 A CN 202110299442A CN 113095154 A CN113095154 A CN 113095154A
- Authority
- CN
- China
- Prior art keywords
- millimeter wave
- wave radar
- image
- data
- dimensional
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 55
- 238000000034 method Methods 0.000 title claims description 15
- 230000004927 fusion Effects 0.000 claims abstract description 10
- 238000000605 extraction Methods 0.000 claims abstract description 9
- 238000010586 diagram Methods 0.000 claims description 17
- 238000004364 calculation method Methods 0.000 claims description 10
- 230000009466 transformation Effects 0.000 claims description 10
- 239000011159 matrix material Substances 0.000 claims description 9
- 239000002184 metal Substances 0.000 claims description 7
- 238000012549 training Methods 0.000 claims description 6
- 238000013528 artificial neural network Methods 0.000 claims description 4
- 238000011176 pooling Methods 0.000 claims description 4
- 238000005070 sampling Methods 0.000 claims description 3
- 235000004522 Pentaglottis sempervirens Nutrition 0.000 claims description 2
- 150000001875 compounds Chemical class 0.000 claims description 2
- 230000000875 corresponding effect Effects 0.000 claims 4
- 230000002596 correlated effect Effects 0.000 claims 1
- 238000009434 installation Methods 0.000 claims 1
- 230000000694 effects Effects 0.000 abstract description 3
- 230000000295 complement effect Effects 0.000 abstract description 2
- 230000006870 function Effects 0.000 description 9
- 238000013135 deep learning Methods 0.000 description 3
- 230000007613 environmental effect Effects 0.000 description 2
- 230000004931 aggregating effect Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000007500 overflow downdraw method Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/64—Three-dimensional objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/07—Target detection
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Biophysics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Multimedia (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Evolutionary Biology (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Radar Systems Or Details Thereof (AREA)
Abstract
The system comprises a space-time alignment module of image data and millimeter wave radar data, a feature extraction module of the image data and the millimeter wave radar data and a fusion detection module of the image data and the millimeter wave radar data; the invention is a fusion target detection method based on two sensors, namely a millimeter wave radar sensor and a monocular camera, and compared with a target detection algorithm based on three-dimensional laser, the cost is lower. The three-dimensional target detection method has the advantages that a good three-dimensional target detection effect can be achieved, the millimeter wave radar can provide accurate information such as object depth and speed, the camera can provide accurate object type information, and the advantages and the disadvantages of the two sensors are complementary.
Description
Technical Field
The invention belongs to the technical field of computer vision combined with deep learning, and particularly relates to a three-dimensional target detection system and method based on a millimeter wave radar and a monocular camera.
Background
The environmental perception is a key part of the unmanned system, is input at the front end of the planning decision module, and provides an important basis for the planning decision module. Environmental awareness includes a variety of tasks including traffic light detection, obstacle detection, lane line detection, etc., where three-dimensional object detection is particularly important. The unmanned vehicle needs to judge whether obstacles exist in the surrounding environment, so that planning decision is made, and the future driving track of the unmanned vehicle is determined. Most autonomous vehicles utilize sensors such as cameras, radars and lidar, the use of different types of sensors has advantages in tasks such as object detection and may result in more accurate and reliable detection, but also makes designing real-time sensing systems more challenging.
Although millimeter wave radar can provide accurate distance and speed information of a target, it is not suitable for tasks such as target classification. On the other hand, a camera is a very effective object classification sensor, but it cannot provide accurate information of object speed and depth. This makes the fusion of millimeter wave radar and image data a very interesting topic in autonomous driving applications.
With the rise of large-scale parallel computing and GPU, the method based on the deep neural network is applied in the field of target detection of the unmanned platform on a large scale. However, due to the lack of an open-source millimeter wave radar data set, methods for fusion of millimeter wave radar and image data based on deep learning are still few.
Disclosure of Invention
The invention aims to provide a three-dimensional target detection system and a three-dimensional target detection method based on a millimeter wave radar and a monocular camera, so as to solve the problems.
In order to achieve the purpose, the invention adopts the following technical scheme: (finalization and replenishment)
Compared with the prior art, the invention has the following technical effects:
the invention is a fusion target detection method based on two sensors, namely a millimeter wave radar sensor and a monocular camera, and compared with a target detection algorithm based on three-dimensional laser, the cost is lower. The method has the advantages that a good three-dimensional target detection effect can be achieved, the millimeter wave radar can provide accurate information such as object depth and speed, the camera can provide rich appearance characteristics including accurate object category information but not a good information source for depth estimation, and advantages and disadvantages of the two sensors are complementary.
The invention relates to a time-space alignment, feature extraction and fusion detection module based on a millimeter wave radar and a monocular camera, which is characterized in that the detection result of the radar is associated with the primary detection result obtained from an image, then the feature extraction is carried out on the radar data, and the three-dimensional detection frame of a target is accurately estimated by combining the image features, so that the accurate detection of an obstacle in an unmanned scene is realized. We feature-extract the image and the millimeter-wave radar data using centret and PointNet, respectively, and accurately estimate three-dimensional properties of the object, such as depth, rotation, and speed information, using the fused features. The existing millimeter wave radar and camera fused deep learning detection network is improved, and the performance is remarkably improved.
Drawings
FIG. 1 is a schematic diagram of a calibration board for a millimeter wave radar and camera according to the present invention;
FIG. 2 is a schematic structural diagram of a fusion detection network of a millimeter wave radar and a camera according to the present invention;
FIG. 3 is a schematic diagram of a data association module of the millimeter wave radar and the camera according to the present invention;
FIG. 4 is a schematic diagram of a feature extraction module of a millimeter wave radar point according to the present invention;
Detailed Description
The invention is further described below with reference to the accompanying drawings:
referring to fig. 1 to 4, first, the image and the millimeter wave radar data are aligned in time and space.
The two sensors acquire data and also acquire time stamps, and in order to ensure the reliability of the data, the data of the millimeter wave radar is aligned to the time stamps of the images. Matching data of one frame of millimeter wave radar to each frame of image data by using nearest neighbor algorithm
Wherein
Is tradarThe target position detected by the radar at the moment, rcs is the radar scattering area, (v)x；vy) Is the radial velocity of the detected object in the x and y directions. Because the millimeter wave radar can provide accurate speed information, the target position is compensated by using the target speed obtained by the detection of the millimeter wave radar, and the calculation process is shown as the following formula:
in the above formula timageIs the acquisition time of the image.
After the time alignment of the two sensors is realized, the fused data of the radar and the image data can be finally obtained. The data of the two sensors that have been aligned in time are then spatially aligned.
It is assumed that the millimeter wave radar detects an obstacle on a plane parallel to the ground, which is a radar plane. Thus, data of the millimeter wave radar can be expressed as ZradarThe coordinate system is a two-dimensional top-view coordinate system, where (x, y) is the position of the target relative to the millimeter wave sensor. And image result level data zimgAnd (u, v) is a two-dimensional image coordinate system, wherein (u, v) is the coordinates of the object in the image coordinate system.
The spatial alignment is to establish projection transformation between a two-dimensional overlook coordinate system where millimeter-wave radar point cloud data is located and an image coordinate system. The mapping relationship between the millimeter wave radar data and the image is shown as follows:
wherein
Can be obtained from the above formula
Referring to fig. 1, a schematic diagram of a millimeter wave radar and an image calibration board according to the present invention is shown. The calculation of the transformation matrix of the two sensor coordinate systems is realized by manufacturing a calibration plate. The calibration plate is a plate pasted with four circular patterns, a metal three-surface corner reflector is placed in the middle of the four circular patterns, the height of the metal three-surface corner reflector from the lower edge of the calibration plate is the same as the height of the millimeter wave radar, the metal three-surface corner reflector can be detected by the millimeter wave radar, and the detection result is in the same plane. The positions of four circles in the acquired image are detected by Hough detection, pixel coordinate values of the four circles are obtained, and finally the pixel coordinate values of the middle of the four circles in the image (namely the metal trihedral corner reflector) are obtained through calculation.
The calibration plate is vertically placed for multiple times at different spatial arrangements and distances, and then multiple groups of different millimeter wave radar detection results (x, y) and corresponding image detection results (u, v) can be obtained. Then, four pairs (or more than four pairs) of (x, y) and (u, v) are manually selected, and a transformation matrix of the millimeter wave radar plane coordinate system and the image coordinate system can be obtained. The solving process is as follows:
from a pair of pairs of characteristic points, the following system of linear equations can be obtained:
since the homography matrix includes the constraint of 1, at least four pairs of corresponding points are needed to calculate the homography matrix. In a real application scene, the homography matrix is calculated by using more than four points, so that the calculation result is more accurate.
And finally projecting the data of the millimeter wave radar after time alignment to an image coordinate system through a transformation matrix.
After millimeter wave radar and camera fused data are obtained through space-time alignment, a feature level fusion method is used for detecting a three-dimensional target.
Fig. 2 is a schematic structural diagram of the fusion detection network of the millimeter wave radar and the camera according to the present invention. Using the DLA network as a backbone network, image features are extracted for predicting the center point of an object on the image, as well as the size of a two-dimensional rectangular frame, the center offset, the size of a three-dimensional rectangular frame, depth information, and rotation information of the object on the image. These values are predicted from the basic regression networks shown in FIG. 2, each consisting of a 3X 3 convolutional layer with 256 dimensions and a 1X 1 convolutional layer. This may provide an accurate two-dimensional rectangular frame and a rough three-dimensional rectangular frame for the objects in the image.
Firstly, adopting a CenterNet algorithm to predict the center point of an image target.
The CenterNet model uses I ∈ RW×H×3As an input to the network, and finally generates one
As an output. W and the slice are the width and the height of the image, R is the down-sampling rate, and C is the number of object types. One is
The output of (C) indicates that an object of type C located at (x, y) in one image is detected. Thermodynamic diagram
Generated from a two-dimensional truth box by a gaussian kernel. Finally, for the position q epsilon R in the image2The value Y of class c of (a) is defined as:
wherein sigmaiThe size of the thermodynamic diagram is controlled based on the size of each object for scale-adaptive standard deviation. The neural network may generate a thermodynamic map for each target type in the image, with the peak representing the likely center point of the object. In the network training process, in order to solve the problem of uneven distribution of positive and negative samples, the Loss function is based on Focal local. The training strategy is to reduce the loss of other pixel points around the central point, and the calculation mode is shown as the following formula:
wherein N is the number of objects,
for a true thermodynamic diagram of the object to be labeled, α and β are the over-parameters of focal loss, and finally α is 2 and β is 4.
For each centroid, the network also predicts a local offset
To compensate for the discretized error caused by the fixed output step in the backbone network, the offset penalty is calculated by the L1 penalty function, as shown in the following equation:
the value of the partial loss function is only at the position of the key point
The above calculation is performed and other positions are ignored.
Class ckCan be enclosed in a two-dimensional rectangular frame of the object kIs represented by (x)1 (k)，y1 (k)，x2 (k)，y2 (k)) The center point of which is at the position
By using
All the center points are predicted. And for each object k returns the size s of its objectk＝(x2 (k)-x1 (k)，y2 (k)-y1 (k)) Use of
All classes of objects are predicted. Calculated using the L1 loss function as follows:
the overall loss function for network training is:
Ldet＝Lk+λsizeLsize+λoffLoff.
in the experiment, λ was setsizeIs 0.1, lambdaoffIs 1.
To generate a three-dimensional rectangular box, the network also regresses three additional attributes, namely depth, angle and size information of the three-dimensional box. Thus, three separate regression network branches were added, the loss values of which were still calculated by the L1 loss function.
It is next necessary to accurately correlate the targets detected by the millimeter wave radar. The centret generates a thermodynamic diagram for each target class in the image. The peaks in the thermodynamic diagram represent the likely center points of the object, and the image features at these locations are used to estimate the properties of other objects. To utilize radar information in such an arrangement, radar-based features need to be mapped to the center of their corresponding object on the image, which requires precise correlation between the targets detected by the radar and the targets in the image.
Fig. 3 is a schematic diagram of a data association module of the millimeter wave radar and the camera according to the present invention. A method based on cone map correlation is used which can significantly reduce the effort for checking the radar data for which a correlation is required, since any radar points beyond this cone are negligible.
In the training phase of the neural network, a region-of-interest viewing cone is created using the three-dimensional truth rectangle of the target and radar detections are associated therewith. In the testing phase, the viewing cones are created by a two-dimensional rectangular box of image predictions and their estimated depth d and size. As shown in FIG. 3, the use of the cone method avoids this problem of associating overlapping objects in two dimensions because the objects do not overlap in three dimensions, and each object will have a separate cone.
Referring to fig. 4, a network structure for extracting millimeter wave radar data according to the present invention is shown. And extracting the characteristics of the millimeter wave radar point cloud data which is well associated with the image through PointNet. Millimeter-wave radars typically represent detected objects as two-dimensional points in a bird's eye view, and may provide their azimuthal and radial distances. For each detection result, the millimeter wave radar can obtain the radial instantaneous speed of the object.
Each radar detection result is represented as a three-dimensional point in the coordinate system of the self-vehicle and is parameterized as P ═ x; y; z; rcs; v ═ yx；vy) One point cloud is represented as a set of three-dimensional points { P }i1., n } for each point PiIs a compound containing (x, y, z, rcs, v)x，vy) The vector of (2). PointNet takes these n points as input, aggregating the characteristics of the points by maximal pooling.
Can be described as a function of a transformation that approximates the elements in a point set, as shown by:
f({x1，...，xn})≈g(h(x1)，...，h(xn))，
and (3) approximating the function h through a multilayer perceptron, approximating the function g through a maximum pooling layer, and finally obtaining the 1024-dimensional global feature. After the global point cloud characteristic corresponding to each viewing cone is calculated, the global characteristic obtained after PointNet processing and the data characteristic of each original radar point are connected to form the characteristic of each radar point, and therefore the obtained characteristic learns the local information and the global information.
The generated millimeter wave radar features are connected together as additional channels and image features. These features are used as input to a second regression to calculate more accurate object information. The velocity regression network estimates the x, y components of the actual velocity of the object in the vehicle coordinate system. The attribute regression network estimates different attributes for different objects, such as Car-like motion or parking, etc. And finally, decoding the regression result into a three-dimensional obstacle frame, and simultaneously obtaining other attributes of the object.
Claims (8)
1. The three-dimensional target detection system based on the millimeter wave radar and the monocular camera is characterized by comprising a space-time alignment module of image data and millimeter wave radar data, a feature extraction module of the image data and the millimeter wave radar data and a fusion detection module of the image data and the millimeter wave radar data;
the space-time alignment module of the image data and the millimeter wave radar data comprises a time alignment module and a space alignment module; the time alignment module is used for aligning the data of the millimeter wave radar to the time stamp of the image data and compensating the position of the image data by using the target speed of the millimeter wave radar; the space alignment module performs spatial association on data of the millimeter wave radar points and corresponding areas of the images through projection transformation;
the feature extraction module of the image data and the millimeter wave radar data is used for extracting image features and extracting features of the millimeter wave radar data; predicting the central point of the target by using the image characteristics to obtain the preliminary estimation of the three-dimensional coordinate and the depth information of the target; performing data association on the millimeter wave radar point and the detected central point of the image by using a method based on a viewing cone, and further performing global feature extraction on the associated millimeter wave radar point cloud data;
and the fusion detection module of the image data and the millimeter wave radar data is used for performing accurate regression of the speed, the depth, the rotation and the attribute information of the three-dimensional object at the second stage after splicing the features extracted by the two sensors.
2. The three-dimensional target detection method based on the millimeter wave radar and the monocular camera is characterized in that the three-dimensional target detection system based on the millimeter wave radar and the monocular camera in claim 1 comprises the following steps:
step 1, performing time-space alignment on an image and millimeter wave radar data, and firstly obtaining a frame of data which comprises a millimeter wave radar and an image and is aligned in time and space;
step 2, after a frame of data containing the millimeter wave radar and the images with aligned time spaces is obtained, feature extraction is respectively carried out on the images and the millimeter wave radar data, and regression of speed, depth, rotation and attribute information of the three-dimensional object in the first stage is carried out by using the features extracted by the images;
and 3, after the extracted millimeter wave radar and the image data features are spliced, performing regression of the speed, the depth, the rotation and the attribute information of the three-dimensional object at the second stage to calculate more accurate object information.
3. The three-dimensional target detection method based on the millimeter wave radar and the monocular camera as claimed in claim 2, wherein in step 1, with the sampling rate of the camera as a reference, the camera selects a frame of data cached by the millimeter wave radar by using a nearest neighbor algorithm every time the camera acquires a frame of image, and then performs position compensation on the frame of data by using accurate speed information provided by the millimeter wave radar so as to synchronize the millimeter wave radar data and the camera data in time; and calculating a coordinate transformation matrix between a plane coordinate system and an image coordinate system of the millimeter wave radar in an off-line manner through a calibration plate, projecting the data of the millimeter wave radar after time alignment to a corresponding image plane through the transformation matrix obtained through calculation, and unifying the detection results of the millimeter wave radar and the monocular camera to the same coordinate system.
4. The millimeter wave radar and monocular camera-based three-dimensional target detection method according to claim 3, wherein the calibration plate is a plate to which four circular patterns are attached, a metal three-dimensional corner reflector is placed in the middle of the four circular patterns, the height of the metal three-dimensional corner reflector from the lower edge of the calibration plate is the same as the installation height of the millimeter wave radar, so as to ensure that the metal three-dimensional corner reflector is detected by the millimeter wave radar and the detection results are all in the same plane, the positions of four circles in the acquired image are detected by using Hough detection, pixel coordinate values of the four circles are obtained, and finally, the pixel coordinate value in the middle of the four circles in the image is obtained by calculation; and vertically placing the calibration plate for multiple times at different spatial arrangements and distances to obtain multiple groups of different millimeter wave radar detection results and corresponding image detection results, and then manually selecting four or more pairs of millimeter wave radar detection results and corresponding image detection results to obtain a transformation matrix of a millimeter wave radar plane coordinate system and an image coordinate system.
5. The method for detecting the three-dimensional target based on the millimeter wave radar and the monocular camera as recited in claim 2, wherein the step 2, the extracting the feature based on the data containing the millimeter wave radar and the image with the time and the space aligned comprises: predicting a central point of an object on an image plane by using a CenterNet model, and then regressing to obtain a primary three-dimensional position, direction and size and object attribute information, wherein the object attribute information comprises the running or parking of a vehicle and the standing or sitting of a pedestrian; and a three-dimensional target region of interest is created by using the two-dimensional rectangular frame predicted by the image and the estimated depth and size of the two-dimensional rectangular frame, radar detection results are associated with the three-dimensional target region of interest, and the point cloud data of the associated local millimeter waves are subjected to global feature extraction by using PointNet.
6. The method for detecting the three-dimensional target based on the millimeter wave radar and the monocular camera of claim 5, wherein the center point of the target in the image is predicted by adopting a CenterNet algorithm:
the CenterNet model uses I ∈ RW×H×3As an input to the network, and finally generates one
As an output; w and H are the width and height of the image, R is the down-sampling rate, and C is the number of object types; one is
The output of (C) indicates that an object of type C located in (x, y) in one image is detected; truth value of thermodynamic diagram
Generating from a two-dimensional truth box by a Gaussian kernel; finally, for the position q epsilon R in the image2The value Y of class c of (a) is defined as:
wherein sigmaiControlling the size of the thermodynamic diagram based on the size of each object for scale-adaptive standard deviation; the neural network generates a thermodynamic diagram for each target type in the image, and the peak value of the thermodynamic diagram represents a possible central point of the object; the training strategy is to reduce the loss of other pixel points around the central point, and the calculation mode is shown as the following formula:
wherein N is the number of objects,
a truth thermodynamic diagram of the object being labeled, α and β being the over-parameters of focal loss;
for each centroid, the network predicts a local offset
To compensate for the discretized error caused by the fixed output step in the backbone network, the offset penalty is calculated by the L1 penalty function, as shown in the following equation:
the value of the partial loss function is only at the position of the key point
The calculation is carried out, and other positions are ignored;
class ckCan be represented as (x)1 (k)，y1 (k)，x2 (k)，y2 (k)) The center point of which is at the position
By using
To predict all center points; and for each object k returns the size s of its objectk＝(x2 (k)-x1 (k)，y2 (k)-y1 (k)) Use of
Predicting objects of all classes; calculated using the L1 loss function as follows:
the overall loss function for network training is:
Ldet＝Lk+λsizeLsize+λoffLoff。
7. the millimeter wave radar and monocular camera-based three-dimensional target detection method according to claim 5, wherein the feature of the millimeter wave radar point cloud data that has been correlated with the image is extracted by PointNet, the millimeter wave radar generally represents the detected object as a two-dimensional point in a bird's eye view and provides its azimuth angle and radial distance, and for each detection result, the millimeter wave radar can obtain the radial instantaneous velocity of the object; each radar detection result is represented as a three-dimensional point in the coordinate system of the self-vehicle and is parameterized as P ═ x; y; z; rcs; v ═ yx；vy) One point cloud is represented as a set of three-dimensional points { P }i1., n } for each point PiIs a compound containing (x, y, z, rcs, v)x，vy) The vector of (a); the PointNet takes the n points as input and aggregates the characteristics of the points through maximum pooling;
described as a function of the transformation that approximates the elements in a point set, as shown by:
f({x1，...，xn})≈g(h(x1)，...，h(xn))，
and (4) approximating the function h through a multilayer perceptron, approximating the function g through a maximum pooling layer, and finally obtaining the global feature.
8. The method as claimed in claim 2, wherein step 3 specifically includes generating a feature map of the millimeter wave radar, connecting the feature map as an additional channel to the image feature, and using the feature map with the radar and image data features as an input of a second regression network of the network, so as to recalculate depth, rotation, speed and attribute information of the object and obtain an accurate result based on the image detection result.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110299442.0A CN113095154A (en) | 2021-03-19 | 2021-03-19 | Three-dimensional target detection system and method based on millimeter wave radar and monocular camera |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110299442.0A CN113095154A (en) | 2021-03-19 | 2021-03-19 | Three-dimensional target detection system and method based on millimeter wave radar and monocular camera |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113095154A true CN113095154A (en) | 2021-07-09 |
Family
ID=76668779
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110299442.0A Pending CN113095154A (en) | 2021-03-19 | 2021-03-19 | Three-dimensional target detection system and method based on millimeter wave radar and monocular camera |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113095154A (en) |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113625271A (en) * | 2021-07-29 | 2021-11-09 | 中汽创智科技有限公司 | Millimeter wave radar and binocular camera based simultaneous positioning and image building method |
| CN114460552A (en) * | 2022-01-21 | 2022-05-10 | 苏州皓宇云联科技有限公司 | Road-end multi-sensor combined calibration method based on high-precision map |
| CN114708585A (en) * | 2022-04-15 | 2022-07-05 | 电子科技大学 | Three-dimensional target detection method based on attention mechanism and integrating millimeter wave radar with vision |
| CN115421132A (en) * | 2022-11-04 | 2022-12-02 | 北京锐达仪表有限公司 | 3D scanning radar with installation angle error is from correcting function |
| CN115797463A (en) * | 2022-11-28 | 2023-03-14 | 湖南华诺星空电子技术有限公司 | Neural network training method and global calibration method for FOD radar and camera |
| CN116386016A (en) * | 2023-05-22 | 2023-07-04 | 杭州睿影科技有限公司 | Foreign matter treatment method and device, electronic equipment and storage medium |
| WO2023164845A1 (en) * | 2022-03-02 | 2023-09-07 | 深圳市大疆创新科技有限公司 | Three-dimensional reconstruction method, device, system, and storage medium |
| WO2024139375A1 (en) * | 2022-12-30 | 2024-07-04 | 华为技术有限公司 | Data processing method and computer device |
| CN118411499A (en) * | 2024-06-21 | 2024-07-30 | 西湖大学 | Target detection method based on tetrahedron model, clustered robot and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111352112A (en) * | 2020-05-08 | 2020-06-30 | 泉州装备制造研究所 | Target detection method based on vision, laser radar and millimeter wave radar |
| CN111427020A (en) * | 2020-06-11 | 2020-07-17 | 交通运输部公路科学研究所 | Combined calibration method, device and system for environmental information data acquisition equipment |
| CN111462237A (en) * | 2020-04-03 | 2020-07-28 | 清华大学 | Target distance detection method for constructing four-channel virtual image by using multi-source information |
| CN111967525A (en) * | 2020-08-20 | 2020-11-20 | 广州小鹏汽车科技有限公司 | Data processing method and device, server and storage medium |
| CN112241978A (en) * | 2020-10-21 | 2021-01-19 | 广州小鹏自动驾驶科技有限公司 | Data processing method and device |
-
2021
- 2021-03-19 CN CN202110299442.0A patent/CN113095154A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111462237A (en) * | 2020-04-03 | 2020-07-28 | 清华大学 | Target distance detection method for constructing four-channel virtual image by using multi-source information |
| CN111352112A (en) * | 2020-05-08 | 2020-06-30 | 泉州装备制造研究所 | Target detection method based on vision, laser radar and millimeter wave radar |
| CN111427020A (en) * | 2020-06-11 | 2020-07-17 | 交通运输部公路科学研究所 | Combined calibration method, device and system for environmental information data acquisition equipment |
| CN111967525A (en) * | 2020-08-20 | 2020-11-20 | 广州小鹏汽车科技有限公司 | Data processing method and device, server and storage medium |
| CN112241978A (en) * | 2020-10-21 | 2021-01-19 | 广州小鹏自动驾驶科技有限公司 | Data processing method and device |
Non-Patent Citations (8)
| Title |
|---|
| CHARLES R. QI ETAL.: "PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation", 《HTTP://ARXIV:1612.00593V2》 * |
| CHARLES R. QI ETAL.: "PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation", 《HTTP://ARXIV:1612.00593V2》, 10 April 2017 (2017-04-10), pages 2 * |
| RAMIN NABATI ETAL.: "CenterFusion: Center-based Radar and Camera Fusion for 3D Object Detection", 《HTTP://ARXIV:2011.04841V1》 * |
| RAMIN NABATI ETAL.: "CenterFusion: Center-based Radar and Camera Fusion for 3D Object Detection", 《HTTP://ARXIV:2011.04841V1》, 10 November 2020 (2020-11-10), pages 1 - 4 * |
| XINGYI ZHOU ETAL.: "Object as Points", 《HTTP://ARXIV:1904.07850V2》 * |
| XINGYI ZHOU ETAL.: "Object as Points", 《HTTP://ARXIV:1904.07850V2》, 25 April 2019 (2019-04-25), pages 3 * |
| 李梦洁: "基于激光点云与图像结合的行人检测与跟踪技术研究", 《中国优秀硕士学位论文全文数据库 (信息科技辑)》, vol. 2020, no. 02, pages 136 - 1943 * |
| 王飞等: "一种雷达与摄像机的空间标定方法", 《计算机测量与控制》, vol. 20, no. 02, pages 454 - 463 * |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113625271A (en) * | 2021-07-29 | 2021-11-09 | 中汽创智科技有限公司 | Millimeter wave radar and binocular camera based simultaneous positioning and image building method |
| CN113625271B (en) * | 2021-07-29 | 2023-10-27 | 中汽创智科技有限公司 | Simultaneous positioning and mapping method based on millimeter wave radar and binocular camera |
| CN114460552A (en) * | 2022-01-21 | 2022-05-10 | 苏州皓宇云联科技有限公司 | Road-end multi-sensor combined calibration method based on high-precision map |
| WO2023164845A1 (en) * | 2022-03-02 | 2023-09-07 | 深圳市大疆创新科技有限公司 | Three-dimensional reconstruction method, device, system, and storage medium |
| CN114708585A (en) * | 2022-04-15 | 2022-07-05 | 电子科技大学 | Three-dimensional target detection method based on attention mechanism and integrating millimeter wave radar with vision |
| CN114708585B (en) * | 2022-04-15 | 2023-10-10 | 电子科技大学 | Attention mechanism-based millimeter wave radar and vision fusion three-dimensional target detection method |
| CN115421132A (en) * | 2022-11-04 | 2022-12-02 | 北京锐达仪表有限公司 | 3D scanning radar with installation angle error is from correcting function |
| CN115421132B (en) * | 2022-11-04 | 2023-02-21 | 北京锐达仪表有限公司 | 3D scanning radar with installation angle error is from correcting function |
| CN115797463A (en) * | 2022-11-28 | 2023-03-14 | 湖南华诺星空电子技术有限公司 | Neural network training method and global calibration method for FOD radar and camera |
| WO2024139375A1 (en) * | 2022-12-30 | 2024-07-04 | 华为技术有限公司 | Data processing method and computer device |
| CN116386016A (en) * | 2023-05-22 | 2023-07-04 | 杭州睿影科技有限公司 | Foreign matter treatment method and device, electronic equipment and storage medium |
| CN116386016B (en) * | 2023-05-22 | 2023-10-10 | 杭州睿影科技有限公司 | Foreign matter treatment method and device, electronic equipment and storage medium |
| CN118411499A (en) * | 2024-06-21 | 2024-07-30 | 西湖大学 | Target detection method based on tetrahedron model, clustered robot and storage medium |
| CN118411499B (en) * | 2024-06-21 | 2024-09-17 | 西湖大学 | Target detection method based on tetrahedron model, clustered robot and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113095154A (en) | Three-dimensional target detection system and method based on millimeter wave radar and monocular camera | |
| US11461912B2 (en) | Gaussian mixture models for temporal depth fusion | |
| Banerjee et al. | Online camera lidar fusion and object detection on hybrid data for autonomous driving | |
| CN111429514A (en) | Laser radar 3D real-time target detection method fusing multi-frame time sequence point clouds | |
| CN112749594B (en) | Information completion method, lane line identification method, intelligent driving method and related products | |
| CN111201451A (en) | Method and device for detecting object in scene based on laser data and radar data of scene | |
| Yang et al. | Reactive obstacle avoidance of monocular quadrotors with online adapted depth prediction network | |
| WO2024114119A1 (en) | Sensor fusion method based on binocular camera guidance | |
| CN111967373B (en) | Self-adaptive enhanced fusion real-time instance segmentation method based on camera and laser radar | |
| Peršić et al. | Online multi-sensor calibration based on moving object tracking | |
| Ji et al. | RGB-D SLAM using vanishing point and door plate information in corridor environment | |
| Liu et al. | Vehicle detection and ranging using two different focal length cameras | |
| Xu et al. | Road boundaries detection based on modified occupancy grid map using millimeter-wave radar | |
| CN117237919A (en) | Intelligent driving sensing method for truck through multi-sensor fusion detection under cross-mode supervised learning | |
| Hara et al. | Vehicle localization based on the detection of line segments from multi-camera images | |
| CN113589848B (en) | Multi-unmanned aerial vehicle detection, positioning and tracking system and method based on machine vision | |
| Ouyang et al. | Semantic slam for mobile robot with human-in-the-loop | |
| CN114898144B (en) | Automatic alignment method based on camera and millimeter wave radar data | |
| CN116977806A (en) | Airport target detection method and system based on millimeter wave radar, laser radar and high-definition array camera | |
| CN116403186A (en) | Automatic driving three-dimensional target detection method based on FPN Swin Transformer and Pointernet++ | |
| CN116385997A (en) | Vehicle-mounted obstacle accurate sensing method, system and storage medium | |
| CN115471526A (en) | Automatic driving target detection and tracking method based on multi-source heterogeneous information fusion | |
| CN114565669A (en) | Method for fusion positioning of field-end multi-camera | |
| Kaida et al. | Behavior prediction using 3d box estimation in road environment | |
| Tu et al. | Method of Using RealSense Camera to Estimate the Depth Map of Any Monocular Camera |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210709 |