PointBeV: A Sparse Approach to BeV Predictions
Loick Chambon, Eloi Zablocki, Mickael Chen, Florent Bartoccioni, Patrick Perez, Matthieu Cord.
Valeo AI, Sorbonne University
|
|
|
| PointBeV reaches state-of-the-art on several segmentation tasks (vehicle without filtering above) while allowing a trade-off between performance and memory consumption. | PointBeV reaches state-of-the-art on several segmentation tasks (vehicle with filtering above). It can also be used using different pattern strategies, for instance a LiDAR pattern. |
|
|
|
| Illustration of different sampling patterns, respectively: a full, a regular, a drivable hdmap, a lane hdmap, a front camera and a LiDAR pattern. PointBeV is flexible to any pattern. | |
We propose PointBeV, a novel sparse BeV segmentation model operating on sparse BeV features instead of dense grids. This approach offers precise control over memory usage, enabling the use of long temporal contexts and accommodating memory-constrained platforms. PointBeV employs an efficient two-pass strategy for training, enabling focused computation on regions of interest. At inference time, it can be used with various memory/performance trade-offs and flexibly adjusts to new specific use cases. PointBeV achieves state-of-the-art results on the nuScenes dataset for vehicle, pedestrian, and lane segmentation, showcasing superior performance in static and temporal settings despite being trained solely with sparse signals.
|
|
| PointBeV architecture is an architecture dealing with sparse representations. It uses an efficient Sparse Feature Pulling module to propagate features from images to BeV and a Sparse Attention module for temporal aggregation. |
If this work is helpful for your research, please consider citing the following BibTeX entry and putting a star on this repository.
@inproceedings{chambon2024pointbev,
title={PointBeV: A Sparse Approach to BeV Predictions},
author={Loick Chambon and Eloi Zablocki and Mickael Chen and Florent Bartoccioni and Patrick Perez and Matthieu Cord},
year={2024},
booktitle={CVPR}
}
- 【28/02/2024】 Code released.
- 【27/02/2024】 PointBeV has been accepted to CVPR 2024.
PointBeV is originally designed for vehicle segmentation. It can be used with different sampling patterns and different memory/performance trade-offs. It can also be used with temporal context to improve the segmentation.
| Models | PointBeV (ours) | BAEFormer | SimpleBeV | BEVFormer | CVT |
|---|---|---|---|---|---|
| IoU | 47.6 | 41.0 | 46.6 | 45.5 | 37.7 |
We also illustrate the results of a temporal model on random samples taken from the NuScenes validation set. The model used for the visualisation is trained without filtering, at resolution 448x800.
PointBeV can be used to perform inference with fewer points than other models. We illustrate this below with a vehicle segmentation model. We can see that PointBeV is able to perform inference with 1/10 of the points used by other models while maintaining a similar performance. This is possible thanks to the sparse approach of PointBeV. In green is represented the sampling mask. Predictions are only performed on the sampled points.
PointBeV can also be used for different segmentation tasks such as pedestrians or hdmap segmentation.
| Models | PointBeV (ours) | TBP-Former | ST-P3 | FIERY | LSS |
|---|---|---|---|---|---|
| IoU | 18.5 | 17.2 | 14.5 | 17.2 | 15.0 |
| Models | PointBeV (ours) | MatrixVT | M2BeV | PeTRv2 | BeVFormer |
|---|---|---|---|---|---|
| IoU | 49.6 | 44.8 | 38.0 | 44.8 | 25.7 |
➡️ Create the environment.
git clone https://github.com/...
cd PointBeV
micromamba create -f environment.yaml -y
micromamba activate pointbev➡️ Install cuda dependencies.
cd pointbev/ops/gs; python setup.py build install; cd -➡️ Datasets.
We used nuScenes dataset for our experiments. You can download it from the official website: https://www.nuscenes.org/nuscenes.
mkdir data
ln -s $PATH/nuscenes data/nuScenes
pytest tests/test_datasets.py➡️ Backbones:
Backbones are downloaded the first time the code is run. We've moved them to a folder so that we can retrieve the weights quickly for other runs.
wget https://download.pytorch.org/models/resnet50-0676ba61.pth -P backbones
wget https://download.pytorch.org/models/resnet101-63fe2227.pth -P backbones
wget https://github.com/lukemelas/EfficientNet-PyTorch/releases/download/1.0/efficientnet-b4-6ed6700e.pth -P backbonesOptional: Preprocess the dataset to train HDmaps model. Building hdmaps 'on the fly' can slow down the dataloader, so we strongly advise you to save the preprocessed dataset.
python pointbev/data/dataset/create_maps.py --split val train --version=trainval
python pointbev/data/dataset/create_maps.py --split mini_val mini_train --version=miniThe directory will be as follows.
< 9929 div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="PointBeV ├── data │ ├── nuScenes │ │ ├── samples │ │ ├── sweeps │ │ ├── v1.0-mini | | ├── v1.0-trainval | |── nuscenes_processed_map | | ├── label | | | ├── mini_train | | | ├── mini_val | | | ├── train | | | ├── val | | ├── map_0.1">PointBeV
├── data
│ ├── nuScenes
│ │ ├── samples
│ │ ├── sweeps
│ │ ├── v1.0-mini
| | ├── v1.0-trainval
| |── nuscenes_processed_map
| | ├── label
| | | ├── mini_train
| | | ├── mini_val
| | | ├── train
| | | ├── val
| | ├── map_0.1