Skip to content

Latest commit

 

History

History
306 lines (255 loc) · 14.8 KB

README.md

File metadata and controls

306 lines (255 loc) · 14.8 KB
Raw

Propose-Reduce VIS

This repo contains the official implementation for the paper:

Video Instance Segmentation with a Propose-Reduce Paradigm

Huaijia Lin*, Ruizheng Wu*, Shu Liu, Jiangbo Lu, Jiaya Jia

ICCV 2021 | Paper

TeaserImage

Installation

Please refer to INSTALL.md.

Demo

You can compute the VIS results for your own videos.

  1. Download a pretrained ResNet-101 and put it in pretrained folder.
mkdir pretrained
  1. Put example videos in 'demo/inputs'. We support two types of inputs, frames directories or .mp4 files (see example for details).
  2. Run the following script and obtain the results in demo/outputs.
sh demo.sh

Data Preparation

(1) Download the videos and jsons of train and val sets from YouTube-VIS 2019

(2) Download the videos and jsons of train and val sets from YouTube-VIS 2021

(3) Download the trainval set of DAVIS-UVOS

(4) Download other pre-computed jsons from data

(5) Symlink the corresponding dataset and json files to the data folder

mkdir data

data
├── trainset_ytv19 --> /path/to/ytv2019/vos/train/JPEGImages/
├── train_ytv19.json --> /path/to/ytv2019/vis/train.json
├── valset_ytv19 --> /path/to/ytv2019/vos/valid/JPEGImages/
├── valid_ytv19.json --> /path/to/ytv2019/vis/valid.json
├── trainset_ytv21 --> /path/to/ytv2021/vis/train/JPEGImages/ 
├── train_ytv21.json --> /path/to/ytv2021/vis/train/instances.json
├── valset_ytv21 --> /path/to/ytv2021/vis/valid/JPEGImages/ 
├── valid_ytv21.json --> /path/to/ytv2021/vis/valid/instances.json
├── trainvalset_davis --> /path/to/DAVIS-UnVOS/DAVIS-trainval/JPEGImages/480p/ 
├── train_davis.json --> /path/to/pre-computed/train_davis.json
├── valid_davis.json --> /path/to/pre-computed/valid_davis.json

Results

We provide the results of several pretrained models and corresponding scripts on different backbones. The results have slight differences from the paper because we make minor modifications to the inference codes.

Download the pretrained models and put them in pretrained folder.

mkdir pretrained
Dataset Method Backbone CA Reduce AP AR@10 download
YouTube-VIS 2019 Seq Mask R-CNN ResNet-50 40.8 49.9 model | scripts
YouTube-VIS 2019 Seq Mask R-CNN ResNet-50 ✓ 42.5 56.8 scripts
YouTube-VIS 2019 Seq Mask R-CNN ResNet-101 43.8 52.7 model | scripts
YouTube-VIS 2019 Seq Mask R-CNN ResNet-101 ✓ 45.2 59.0 scripts
YouTube-VIS 2019 Seq Mask R-CNN ResNeXt-101 47.6 56.7 model | scripts
YouTube-VIS 2019 Seq Mask R-CNN ResNeXt-101 ✓ 48.8 62.2 scripts
YouTube-VIS 2021 Seq Mask R-CNN ResNet-50 39.6 47.5 model | scripts
YouTube-VIS 2021 Seq Mask R-CNN ResNet-50 ✓ 41.7 54.9 scripts
YouTube-VIS 2021 Seq Mask R-CNN ResNeXt-101 45.6 52.9 model | scripts
YouTube-VIS 2021 Seq Mask R-CNN ResNeXt-101 ✓ 47.2 57.6 scripts
Dataset Method Backbone J&F J F download
DAVIS-UVOS Seq Mask R-CNN ResNet-101 68.1 64.9 71.4 model | scripts
DAVIS-UVOS Seq Mask R-CNN ResNeXt-101 70.6 67.2 73.9 model | scripts

Evaluation

YouTube-VIS 2019: A json file will be saved in ../Results_ytv19 folder. Please zip and upload to the codalab server.

YouTube-VIS 2021: A json file will be saved in ../Results_ytv21 folder. Please zip and upload to the codalab server.

DAVIS-UVOS: Color masks will be saved in ../Results_davis folder. Please use the official code for evaluation.

Training

To reproduce the results, we provide the pre-trained model on the main-training stage and the training scripts for the finetuning stage (described in Sec. 4.2 of the paper).

Please put the pre-trained model into pretrained folder and then run the corresponding script.

Dataset Method Backbone download
YouTube-VIS 2019 Seq Mask R-CNN ResNet-50 model | scripts
YouTube-VIS 2019 Seq Mask R-CNN ResNet-101 model | scripts
YouTube-VIS 2019 Seq Mask R-CNN ResNeXt-101 model | scripts
YouTube-VIS 2021 Seq Mask R-CNN ResNet-50 model | scripts
YouTube-VIS 2021 Seq Mask R-CNN ResNeXt-101 model | scripts
DAVIS-UVOS Seq Mask R-CNN ResNet-101 model | scripts
DAVIS-UVOS Seq Mask R-CNN ResNeXt-101 model | scripts

The trained checkpoints will be saved in ../work_dirs folder. To evaluate the effect, please replace the pretrained weights of inference with the trained checkpoints and run the inference scripts.

TODOs

Citation

If you find this work useful in your research, please cite:

@article{lin2021video,
  title={Video Instance Segmentation with a Propose-Reduce Paradigm},
  author={Lin, Huaijia and Wu, Ruizheng and Liu, Shu and Lu, Jiangbo and Jia, Jiaya},
  booktitle={IEEE International Conference on Computer Vision (ICCV)},
  year={2021}
}

Contact

If you have any questions regarding the repo, please feel free to contact me ([email protected]) or create an issue.

Acknowledgments

This repo is based on MMDetection, MaskTrackRCNN, STM, MMCV and COCOAPI.