SceneTracker: Long-term Scene Flow Estimation Network

This repository contains the source code for our paper:

• SceneTracker: Long-term Scene Flow Estimation Network | Paper | 中文介绍
  • To better capture the fine-grained and long-term 3D motion, a comprehensive new task, long-term scene flow estimation (LSFE), is studied.
  • A novel learning-based LSFE network, SceneTracker, is presented. SceneTracker shows superior capabilities in handling 3D spatial occlusion and depth noise interference.
  • We build the first real-world evaluation dataset, LSFDriving, further substantiating SceneTracker's commendable generalization capacity. Specifically, the tracked points are sampled from static backgrounds, moving rigid vehicles, and moving non-rigid pedestrians' joints and garments.

Updates

• [2024.07.12] 📣 A demo of SceneTracker is released!
• [2024.07.04] 📣 The code and trained weights of SceneTracker are released!
• [2024.05.30] 📣 The data of LSFOdyssey is released!
• [2024.04.28] 📣 The data of LSFDriving is coming soon!
• [2024.03.29] 📣 The paper of SceneTracker is made public!

Environment

Our code has been successfully tested in the following environments:

• NVIDIA 3090 GPU
• CUDA 11.1
• Python 3.8
• PyTorch 1.8.2

conda create -n scenetracker python=3.8
conda activate scenetracker

pip install torch==1.8.2 --extra-index-url https://download.pytorch.org/whl/lts/1.8/cu111
pip install einops==0.4.1
pip install pillow==9.5.0
pip install opencv-python==4.9.0.80
pip install albumentations==1.3.1
pip install timm==0.9.12

Trained Weights

Download the weights below and put them in the exp/0-pretrain path.

Model	Training process	Weights	Comments
SceneTracker	Odyssey	scenetracker_odyssey_200k.pth Huggingface & BaiduNetdisk	Best performance on LSFOdyssey

Demo

• Quick start.

  bash script/demo.sh

Datasets

To train / test SceneTracker, you will need to download the proposed datasets and update data_root in data/dataset.py.

• LSFOdyssey: Huggingface | BaiduNetdisk
• LSFDriving

Training

• Train SceneTracker under the Odyssey training process.

  bash script/train_odyssey.sh

Testing

• Test SceneTracker on LSFOdyssey.

  bash script/test_odyssey.sh

Acknowledgments

We would like to thank CoTracker, PointOdyssey and SplatFlow for publicly releasing their code and data.

Citing this Work

If you find our repository useful, please consider giving it a star ⭐ and citing our paper in your work:

@article{wang2024scenetracker,
  title={SceneTracker: Long-term Scene Flow Estimation Network},
  author={Wang, Bo and Li, Jian and Yu, Yang and Liu, Li and Sun, Zhenping and Hu, Dewen},
  journal={arXiv preprint arXiv:2403.19924},
  year={2024}
}