This is the official code release for
- MS3D: Leveraging Multiple Detectors for Unsupervised Domain Adaptation in 3D Object Detection [paper]
- MS3D++: Ensemble of Experts for Multi-Source Unsupervised Domain Adaptation in 3D Object Detection [paper] [presentation]
MS3D is a simple auto-labeling framework for vehicles and pedestrians that generates high quality labels for training 3D detectors on different lidar types, regardless of their scan pattern or point cloud density.
To auto-label your own point cloud data, see our guide. To obtain the best results, please use multi-sweep pre-trained models for auto-labeling. Use both dense and sparse lidar pre-trained models in the auto-labeling ensemble for the best chance of generalization.
Get a quick understanding of MS3D with our explanation video and tutorial notebook.
Our MS3D framework has the following benefits:
- Robust labeling of a wide range of lidars.
- High performance of 3D detectors trained with MS3D labels, achieving BEV vehicle detection comparable to training with human-annotated labels.
- Tailor the ensemble with any pre-trained 3D detectors to obtain high quality labels on any lidar dataset.
- Train any 3D detector with any data augmentation. The final generated labels can replace human-annotated labels in supervised training of any 3D detector and data augmentation.
- Preserves real-time inference capability of detectors as we don't modify detector architecture.
Our box fusion method, KBF, can be used for detector ensembling in a supervised setting as well and can outperform Weighted Box Fusion (WBF). See our first MS3D paper for comparison results and a simple demo here.
This main branch is the official release for MS3D++ and is built upon OpenPCDet v0.6.0. If you wish to use our previous MS3D code, please refer to the MS3D branch.
Please refer to INSTALL.md for the installation of MS3D.
We provide the following guides to learn how to use MS3D.
- GETTING_STARTED.md to reproduce our paper results.
- AUTO_LABEL_YOUR_DATA.md to auto-label your own point cloud data.
- PARAMETERS.md to tune MS3D config file parameters.
- VISUALIZATION.md to use our visualization tools for label assessment.
We provide a tutorial to demonstrate how MS3D auto-labels a folder of point clouds.
In this section, we provide a collection of pre-trained (aka. off-the-shelf) models that can be used to reproduce our results or label your own datasets. If you wish to download multiple detectors, we provide links to download the entire folder of detectors trained on nuScenes and Lyft to save some time.
Reported 3D average precision is the oracle performance for the range breakdown 0-30m / 30-50m / 50-80m when trained and tested on the same dataset.
| Source | Detector | Sweeps | Vehicle | Pedestrian | Download |
|---|---|---|---|---|---|
| Lyft | PV-RCNN++ (Anchor) | 1 | 88.9 / 68.2 / 25.0 | 55.1 / 26.8 / 11.9 | model |
| Lyft | PV-RCNN++ (Center) | 1 | 87.1 / 66.2 / 23.6 | 50.0 / 27.7 / 10.5 | model |
| Lyft | VoxelRCNN (Anchor) | 1 | 87.8 / 66.3 / 22.5 | 54.5 / 29.8 / 10.8 | model |
| Lyft | VoxelRCNN (Center) | 1 | 88.6 / 66.9 / 22.8 | 52.5 / 27.3 / 11.0 | model |
| Lyft | PV-RCNN++ (Anchor) | 3 | 90.3 / 73.3 / 29.0 | 57.0 / 33.4 / 19.2 | model |
| Lyft | PV-RCNN++ (Center) | 3 | 88.2 / 71.1 / 27.8 | 53.9 / 33.1 / 17.5 | model |
| Lyft | VoxelRCNN (Anchor) | 3 | 88.0 /79.7 / 26.2 | 57.8 / 36.6 / 18.9 | model |
| Lyft | VoxelRCNN (Center) | 3 | 88.0 / 70.4 / 26.3 | 59.5 / 34.4 / 18.9 | model |
| Lyft | IA-SSD | 3 | 82.6 / 58.7 / 17.6 | 28.9 / 18.9 / 12.2 | model |
| nuScenes | PV-RCNN++ (Anchor) | 10 | 72.6 / 20.8 / 2.6 | 44.0 / 13.8 / 1.4 | model |
| nuScenes | PV-RCNN++ (Center) | 10 | 68.9 / 18.9 / 2.2 | 42.2 / 14.8 / 1.4 | model |
| nuScenes | VoxelRCNN (Anchor) | 10 | 69.8 / 17.2 / 2.1 | 42.7 / 12.3 / 0.9 | model |
| nuScenes | VoxelRCNN (Center) | 10 | 66.6 / 17.5 / 1.9 | 43.2 / 14.8 / 1.7 | model |
| nuScenes | IA-SSD | 10 | 57.0 / 10.2 / 0.8 | 31.5 / 8.9 / 0.7 | model |
| Waymo | PV-RCNN++ (Anchor) | 1 | 90.2 / 66.4 / 38.8 | 68.3 / 57.2 / 39.0 | - |
| Waymo | PV-RCNN++ (Center) | 1 | 90.6 / 68.2 / 40.1 | 76.6 / 67.6 / 51.2 | - |
| Waymo | VoxelRCNN (Anchor) | 1 | 89.9 / 65.3 / 37.0 | 67.9 / 56.2 / 36.1 | - |
| Waymo | VoxelRCNN (Center) | 1 | 90.2 / 67.9 / 39.3 | 76.0 / 66.8 / 48.8 | - |
| Waymo | IA-SSD | 1 | 86.7 / 59.7 / 31.3 | 60.9 / 55.4 / 42.6 | - |
| Waymo | PV-RCNN++ (Anchor) | 4 | 90.4 / 68.2 / 40.7 | 67.7 / 56.7 / 37.0 | - |
| Waymo | PV-RCNN++ (Center) | 4 | 91.1 / 70.1 / 42.3 | 75.9 / 68.1 / 53.0 | - |
| Waymo | VoxelRCNN (Anchor) | 4 | 90.8 / 69.8 / 43.6 | 68.9 / 61.0 / 46.6 | - |
| Waymo | VoxelRCNN (Center) | 4 | 91.1 / 71.7 / 45.5 | 78.5 / 71.7 / 60.3 | - |
If you would like to contribute to this table with different models that are trained on different datasets using OpenPCDet, please email me at [email protected] with the cfgs/model and I can add it in. Note that the models should be trained with SHIFT_COOR for better cross-domain performance.
We do not provide links to Waymo models due to the Waymo Dataset License Agreement. If you would like to have the Waymo pre-trained models, please send me an email with your name, institute, a screenshot of the Waymo dataset registration confirmation mail and your intended usage. Note that we are not allowed to share the model with you if it will use for any profit-oriented activities.
In this section we provide the final models after multiple self-training rounds which were used for our paper's results. Average precision results are reported as BEV / 3D with KITTI's evaluation at 40 recall levels.
We also provide the final set of pseudo-labels for each target domain. These can be directly used to train other detectors.
nuScenes was auto-labeled using an ensemble of Waymo and Lyft pre-trained detectors. The final set of pseudo-labels can be downloaded here.
| Method | Detector | Sweeps | Vehicle | Pedestrian | Download |
|---|---|---|---|---|---|
| MS3D | SECOND-IoU | 1 | 42.2 / 24.7 | - | model |
| MS3D++ | SECOND-IoU | 1 | 43.9 / 23.1 | - | model |
| MS3D | VoxelRCNN (Center) | 10 | 49.2 / 27.5 | - | model |
| MS3D++ | VoxelRCNN (Center) | 10 | 50.3 / 27.2 | 25.8 / 15.9 | model |
| MS3D++ | VoxelRCNN (Anchor) | 10 | 52.1 / 26.5 | 27.0 / 15.4 | model |
| GT-Trained | VoxelRCNN (Center) | 10 | 56.4 / 37.2 | 41.7 / 32.5 | model |
| GT-Trained | VoxelRCNN (Anchor) | 10 | 55.3 / 36.6 | 38.3 / 29.7 | model |
Waymo was auto-labeled using an ensemble of nuScenes and Lyft pre-trained detectors. The final set of pseudo-labels can be downloaded here.
| Method | Detector | Sweeps | Vehicle | Pedestrian | Download |
|---|---|---|---|---|---|
| MS3D | VoxelRCNN (Center) | 4 | 64.3 / 47.7 | - | - |
| MS3D++ | VoxelRCNN (Center) | 4 | 70.6 / 52.8 | 57.0 / 51.8 | - |
| MS3D++ | VoxelRCNN (Anchor) | 4 | 70.3 / 52.3 | 52.7 / 48.9 | - |
| GT-Trained | VoxelRCNN (Center) | 4 | 75.1 / 61.2 | 67.8 / 62.9 | - |
| GT-Trained | VoxelRCNN (Anchor) | 4 | 73.8 / 60.5 | 57.8 / 54.7 | - |
As mentioned above regarding Waymo's licensing, email me at [email protected] if you wish to download the Waymo models. You can also train these models yourself using our provided cfg files and pseudo-labels.
Lyft was auto-labeled using an ensemble of Waymo and nuScenes pre-trained detectors. The final set of pseudo-labels can be downloaded here.
| Method | Detector | Sweeps | Vehicle | Pedestrian | Download |
|---|---|---|---|---|---|
| MS3D | VoxelRCNN (Center) | 3 | 77.3 / 63.4 | - | model |
| MS3D++ | VoxelRCNN (Center) | 3 | 77.0 /66.0 | 46.9 / 43.3 | model |
| MS3D++ | VoxelRCNN (Anchor) | 3 | 77.2 / 65.3 | 47.2 / 43.6 | model |
| GT-Trained | VoxelRCNN (Center) | 3 | 86.8 / 74.7 | 60.6 / 54.2 | model |
| GT-Trained | VoxelRCNN (Anchor) | 3 | 85.2 / 72.7 | 58.8 / 50.3 | model |
Take a look at more of our visualizations for MS3D++ qualitative and MS3D qualitative.
MS3D is released under the Apache 2.0 license.
If you find this project useful in your research, please give us a star and consider citing:
@article{tsai2024ms3d++,
author={Tsai, Darren and Berrio, Julie Stephany and Shan, Mao and Nebot, Eduardo and Worrall, Stewart},
journal={IEEE Transactions on Intelligent Vehicles},
title={MS3D++: Ensemble of Experts for Multi-Source Unsupervised Domain Adaptation in 3D Object Detection},
year={2024},
pages={1-16},
doi={10.1109/TIV.2024.3441527}
}
@inproceedings{tsai2023ms3d,
author={Tsai, Darren and Berrio, Julie Stephany and Shan, Mao and Nebot, Eduardo and Worrall, Stewart},
booktitle={2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC)},
title={MS3D: Leveraging Multiple Detectors for Unsupervised Domain Adaptation in 3D Object Detection},
year={2023},
pages={140-147},
doi={10.1109/ITSC57777.2023.10421941}
}