MCTrack: A Unified 3D Multi-Object Tracking Framework for Autonomous

0. Abstract

This paper introduces MCTrack, a new 3D multi-object tracking method that achieves state-of-the-art (SOTA) performance across KITTI, nuScenes, and Waymo datasets. Addressing the gap in existing tracking paradigms, which often perform well on specific datasets but lack generalizability, MCTrack offers a unified solution. Additionally, we have standardized the format of perceptual results across various datasets, termed BaseVersion, facilitating researchers in the field of multi-object tracking (MOT) to concentrate on the core algorithmic development without the undue burden of data preprocessing. Finally, recognizing the limitations of current evaluation metrics, we propose a novel set that assesses motion information output, such as velocity and acceleration, crucial for downstream tasks.

1. News

• [🔥🔥🔥2024-10-08]. The code has been released.🙌
• 2024-09-24. MCTrack is released on arXiv.
• 2024-09-01. We rank 2nd among all methods on Waymo Dataset for MOT.
• 2024-08-30. We rank 1st among all methods on KITTI Dataset for MOT.
• 2024-08-27. We rank 1st among all methods on nuScenes Dataset for MOT.

2. Results

KITTI

online

Method	Detector	Set	HOTA	MOTA	TP	FP	IDSW
MCTrack	VirConv	test	80.78	89.82	32207	2185	64
MCTrack	VirConv	train	82.60	85.61	22107	1468	32

offline

Method	Detector	Set	HOTA	MOTA	TP	FP	IDSW
MCTrack	VirConv	test	82.56	91.64	32064	2328	12
MCTrack	VirConv	train	83.88	86.61	22112	1261	3

nuScenes

Method	Detector	Set	AMOTA	MOTA	TP	FP	IDS
MCTrack	LargeKernel3D	test	0.763	0.634	103327	19643	242
MCTrack	CenterPoint	val	0.740	0.640	85900	13083	275

Waymo

Method	Detector	Set	MOTA / L1	MOTP / L1	MOTA / L2	MOTP / L2
MCTrack	CTRL	test	0.7504	0.2276	0.7344	0.2278
MCTrack	CTRL	val	0.7384	0.2288	0.7155	0.2293

3. Data preparation

BaseVersion Data Generation

• First, you need to download the original datasets from Kitti, nuScenes, and Waymo, as well as their corresponding detection results, and organize them in the following directory structure. (Note: If you only want to test on the KITTI dataset, you only need to download the KITTI data.)
  • For KITTI

    data/
    └── kitti/
        ├── datasets/
        |    ├── testing/
        |    |    ├── calib/
        |    |    |   └── 0000.txt
        |    |    └── pose/
        |    |        └── 0000.txt
        |    └── training/
        |         ├── calib/
        |         ├── label_02/
        |         └── pose/
        └── detectors/
             ├── casa/
             │    ├── testing/
             │    │   ├── 0000/
             │    │   │   └── 000000.txt
             │    │   │   └── 000001.txt             
             │    │   └── 0001/
             │    └── testing/
             └── point_rcnn/
    

  • For nuScenes

    data/
    └── nuScenes/
        ├── datasets/
        |    ├── maps/
        |    ├── samples/
        |    ├── sweeps/
        |    ├── v1.0-test/
        |    └── v1.0-trainval/
        └── detectors/
             ├── centerpoint/
             |   └── val.json
             └── largekernel/
                 └── test.json
    

  • For Waymo

    • To prepare the Waymo data, you first need to follow ImmortalTracker's instructions to extract ego_info and ts_info (we will also provide these in the link, so you might be able to skip this step.).

    • Follow ImmortalTracker's instructions to convert detection results into to .npz files.

    • Please note that we have modified the ego_info section in immortaltracker, and the updated file is provided in preprocess/ego_info.py.

    data/
    └── Waymo/
        ├── datasets/
        |    ├── testing/
        |    |    ├── ego_info/
        |    |    │   ├── .npz
        |    |    │   └── .npz             
        |    |    └── ts_info/
        |    |        ├── .json
        |    |        └── .json          
        |    └── validation/
        |         ├── ego_info/
        |         └── ts_info/
        └── detectors/
             └── ctrl/
                  ├── testing/
                  │   ├── .npz
                  │   └── .npz        
                  └── validation/
                      ├── .npz
                      └── .npz 
    

• Second, run the following command to generate the BaseVersion data format required for MCTrack. Certainly, if you do not wish to regenerate the data, you can directly download the data we have prepared from Google Drive and Baidu Cloud. Due to copyright issues with the Waymo dataset, we are unable to provide the corresponding converted data.

  $ python preprocess/convert2baseversion.py --dataset kitti/nuscenes/waymo
  

• Eventually, you will get the data format of baseversion in the path data/base_version/.

  data/
  └── base_version/
      ├── kitti/
      │   ├── casa/
      │   |   ├── test.json
      │   |   └── val.json
      │   └── virconv/
      │       ├── test.json
      │       └── val.json
      ├── nuscenes/
      |   ├── centerpoint/
      |   │   └── val.json
      |   └── largekernel/
      |        └── test.json
      └── waymo/
          └── ctrl/
              ├── val.json
              └── test.json
  

BaseVersion Data Format

scene-0001/
├── frame_0/
│   ├── cur_sample_token                # for nuScenes
│   ├── timestamp                       # The timestamp of each frame
│   ├── bboxes/                         # Detected bbox
│   │   ├── bbox_1/                     # Bbox1
│   │   │   ├── detection_score         # Detection score
│   │   │   ├── category                # Category
│   │   │   ├── global_xyz              # Center position of the global bbox
│   │   │   ├── global_orientation      # Orientation quaternion
│   │   │   ├── global_yaw              # Yaw
│   │   │   ├── lwh                     # Length, width, and height of the bbox
│   │   │   ├── global_velocity         # Velocity of the object in the global coordinate 
│   │   │   ├── global_acceleration     # Acceleration of the object in the global coordinate 
│   │   │   └── bbox_image/             # Information of the bbox in the image coordinate
│   │   │       ├── camera_type         # Camera position
│   │   │       └── x1y1x2y2            # Image coordinates
│   │   ├── bbox_2/
│   │   │   ├── detection_score
│   │   │   ├── category
│   │   │   └── ...
│   │   └── ...
│   └── transform_matrix/
│       ├── global2ego                 # Transformation matrix from global to ego 
│       ├── ego2lidar                  # Transformation matrix from ego to lidar
│       ├── global2lidar               # Transformation matrix from global to lidar 
│       └── cameras_transform_matrix/  # Camera-related transformation matrix
│           ├── CAM_FRONT/             # Front-view camera
│           │   ├── image_shape        # Image shape
│           │   ├── ego2camera         # Transformation matrix from ego to camera
│           │   ├── camera2image       # Transformation matrix from camera to image
│           │   ├── lidar2camera       # Transformation matrix from lidar to camera
│           │   ├── camera_token       # for nuScenes
│           │   └── camera_path        # for nuScenes
│           ├── CAM_FRONT_RIGHT/
│           │   └── ...
│           └── ...
├── frame_1/
│   └── ...
└── ...

4. Installation

Basic environment configuration

Create virtual environment

$ conda create -n MCTrack python=3.8

Activate virtual environment

$ conda activate MCTrack

Install Python package

$ pip install -r requirements.txt

Dataset configuration

nuScenes and KITTI

• For KITTI and nuScenes, you can run MCTrack directly after installing the required packages as mentioned above.

Waymo

• Please follow the official tutorial to install waymo_open_dataset package

• Use the following command to verify if the installation was successful.

  $ cd /content/waymo-od/src/ && bazel-bin/waymo_open_dataset/metrics/tools/compute_detection_metrics_main waymo_open_dataset/metrics/tools/fake_predictions.bin  waymo_open_dataset/metrics/tools/fake_ground_truths.bin
  

5. Evaluation

Local evaluation

• Run directly:

  $ python main.py --dataset kitti/nuscenes/waymo -e -p 1
  

• For example, if you want to run kitti evaluation:

  $ python main.py --dataset kitti -e -p 1
  

• If you want to run the tracking evaluation faster, you can use multi-processing:

  $ python main.py --dataset kitti -e -p 8
  

• The results are saved in the results folder. You can modify the evaluation parameters in the config/kitti.yaml file. -e represents whether to evaluate the results.

• Note: for waymo dataset, you should modify the waymo_open_dataset package path to your WOD path first

  $ vim evaluation/static_evaluation/waymo/eval.py
  

Submit

KITTI

• If you want to submit the test set results online, you need to change SPLIT: to test and DETECTOR: to virconv in the config/kitti.yaml file. Then, rerun the tracking program to generate the 0000.txt/0001.txt/.../0028.txt files. After that, compress that into a .zip file and submit it to the kitti tracking challenge.

nuScenes

• If you want to submit the test set results online, you need to change SPLIT: to test and DETECTOR: to largekernel in the config/nuscenes.yaml file. Then, rerun the tracking program to generate the result.json file. After that, compress the result.json into a .zip file and submit it to the nuScenes tracking challenge.

Waymo

• Modify the submission file with your information

  $ vim waymo-od/src/waymo_open_dataset/metrics/tools/submission.txtpb
  

• Generate the result

  $ mkdir test_result
  $ waymo-od/src/bazel-bin/waymo_open_dataset/metrics/tools/create_submission  --input_filenames='results/waymo/testing/bin/pred.bin' --output_filename='test_result/model' --submission_filename='waymo-od/src/waymo_open_dataset/metrics/tools/submission.txtpb'
  $ tar cvf test_result/my_model.tar test_result/
  $ gzip test_result/my_model.tar
  

• Submit your result to the waymo tracking challenge.

Motion metric evaluation

• TODO:Currently, we are only conducting motion metric evaluations on the nuScenes dataset.

• If you are interested in our motion metric evaluation, you first need to convert the tracking result files (result_for_motion.json) into a format suitable for motion metric evaluation by running:

  $ python preprocess/motion_dataset/convert_nuscenes_result_to_pkl.py
  

• result_path represents the path where the tracking program saves the results(result_for_motion.json), nusc_path refers to the original path of the nuScenes dataset, and gt_pkl_path, det_pkl_path, kalman_cv_pkl_path, diff_pkl_path and curve_pkl_path represent the data files used for motion metric evaluation.

• Next, run:

  $ python evaluation/eval_motion.py
  

• You will then obtain the results of the motion metric evaluation. The config/nuscenes_motion_eval.yaml file contains the parameters for motion metric evaluation.

6. Acknowledgement

• In the detection part, many thanks to the following open-source projects:

  • CTRL

  • VirConv

  • CenterPoint

• In the tracking part, many thanks to the following open-source projects:

  • PC3T

  • Poly-MOT

  • ImmortalTracker

7. Citation

If you find this work useful, please consider to cite our paper:

@article{wang2024mctrack,
  title={MCTrack: A Unified 3D Multi-Object Tracking Framework for Autonomous Driving},
  author={Wang, Xiyang and Qi, Shouzheng and Zhao, Jieyou and Zhou, Hangning and Zhang, Siyu and Wang, Guoan and Tu, Kai and Guo, Songlin and Zhao, Jianbo and Li, Jian and others},
  journal={arXiv preprint arXiv:2409.16149},
  year={2024}
}