Similarity Contrastive Estimation for Self-Supervised Soft Contrastive Learning (WACV 2023)

News 2: A new repository containing the code of this paper as well as its extensions to video representation learning is available here.

News 1: SCE has been extended to video and the paper is available here.

Introduction

You should now use eztorch, this repo is deprecated.

This repository contains a Pytorch implementation of Similarity Contrastive Estimation for Self-Supervised Soft Contrastive Learning (SCE) that has been published in IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2023.

It relies on several packages to optimize training and simplify the code such as:

• Hydra for configuration.
• Pytorch-Lightning for simplifying the code.
• Kornia to apply augmentations on GPU and speed-up training.

Main results

The following results are the main ones reported in our paper:

pretrain epochs	pretrain crops	linear accuracy	ckpt
100	2x224	72.1	Download
200	2x224	72.7	Download
300	2x224	73.3	Download
1000	2x224	74.1	Download
200	2x224 + 192 + 160 + 128 + 96	75.4	Download

You can find below the command lines to launch the configs to retrieve those results and above the checkpoint links.

How to Install

To install this repository you first need to install a recent version of Pytorch (> 1.8).

Finally launch the following command:

cd sce
pip install -e .

The argument -e makes a dev installation that allows you to make changes in the repository without needing to install the package again. It is optional.

Warning If you want to manually install the dependencies, then, you should install Kornia from the Julien Denize's fork that aligned transformations of Kornia with expected results from Torchvision.

Usage: SCE Pretraining

We launched our experiments on a computational cluster configured via SLURM using for two crops 8 A100-80G GPUs and for multi-crop 16 A100-80G GPUs.

We provide below the commands using the srun command from SLURM that was inside a SLURM script. Pytorch-Lightning directly detects SLURM is used and configure accordingly the distributed training. We strongly suggest you to refer to the Pytorch-Lightning's documentation to correctly set up a command line without srun if you do not have access to a slurm cluster.

Pretraining for 100 epochs

output_dir=...
dataset_dir=...

config_path="../sce/configs/run/pretrain/sce/resnet50"
config_name="resnet50_imagenet"
seed=42

cd sce/run

srun --kill-on-bad-exit=1 python pretrain.py -cp $config_path -cn $config_name\
    dir.data=$dataset_dir dir.root=$output_dir \
    dir.exp='pretrain' seed.seed=$seed \
    datamodule.train.loader.num_workers=8 \
    datamodule.val.loader.num_workers=8 \
    trainer=gpu_ddp_sbn_amp trainer.gpus=8 \
    trainer.max_epochs=100 model.optimizer.initial_lr=0.6

Pretraining for 200 epochs

srun --kill-on-bad-exit=1 python pretrain.py -cp $config_path -cn $config_name\
    dir.data=$dataset_dir dir.root=$output_dir \
    dir.exp='pretrain' seed.seed=$seed \
    datamodule.train.loader.num_workers=8 \
    datamodule.val.loader.num_workers=8 \
    trainer=gpu_ddp_sbn_amp trainer.gpus=8

Pretraining for 300 epochs

srun --kill-on-bad-exit=1 python pretrain.py -cp $config_path -cn $config_name\
    dir.data=$dataset_dir dir.root=$output_dir \
    dir.exp='pretrain' seed.seed=$seed \
    datamodule.train.loader.num_workers=8 \
    datamodule.val.loader.num_workers=8 \
    trainer=gpu_ddp_sbn_amp trainer.gpus=8 \
    trainer.max_epochs=300

Pretraining for 1000 epochs

srun --kill-on-bad-exit=1 python pretrain.py -cp $config_path -cn $config_name\
    dir.data=$dataset_dir dir.root=$output_dir \
    dir.exp='pretrain' seed.seed=$seed \
    datamodule.train.loader.num_workers=8 \
    datamodule.val.loader.num_workers=8 \
    trainer=gpu_ddp_sbn_amp trainer.gpus=8 \
    trainer.max_epochs=1000 \
    model.optimizer.params.weight_decay=1.5e-6

Pretraining for 200 epochs with multi-crop

config_name="resnet50_imagenet_multi_crop"

srun --kill-on-bad-exit=1 python pretrain.py -cp $config_path -cn $config_name \
    dir.data=$dataset_dir dir.root=$output_dir \
    dir.exp='pretrain' seed.seed=$seed \
    datamodule.train.loader.num_workers=8 \
    datamodule.val.loader.num_workers=8 \
    trainer=gpu_ddp_sbn_amp +trainer.num_nodes=2 \
    trainer.gpus=8

Usage: Linear classification

Same as for pretraining we launched our experiment on a SLURM cluster.

eval_config_path="../sce/configs/run/evaluation/linear_classifier/mocov3/resnet50"
eval_config_name="resnet50_imagenet"
checkpoint=...

srun --kill-on-bad-exit=1 python linear_classifier_evaluation.py \
    -cp $eval_config_path -cn $eval_config_name \
    dir.data=$dataset_dir dir.root=$output_dir \
    dir.exp='linear_classifier_evaluation' \
    model.pretrained_trunk_path=$checkpoint seed.seed=$seed \
    datamodule.train.loader.num_workers=8 \
    datamodule.val.loader.num_workers=8 trainer.gpus=8

Usage: Transfer Learning

For Transfer Learning evaluation we used code provided by several authors that we would like to thank for sharing their work. Below we redirect you to their GitHub for every transfer we have done.

All evaluations are based on the multi-crop checkpoint.

Linear classifier on other datasets

To evaluate the transferability of our pretrained checkpoints on various datasets by training a linear classifier, we used the ssl-transfer repository.

Method	Food101	CIFAR10	CIFAR100	SUN397	Cars	Aircraft	VOC2007	DTD	Pets	Caltech101	Flowers	Avg.
SimCLR	72.8	90.5	74.4	60.6	49.3	49.8	81.4	75.7	84.6	89.3	92.6	74.6
BYOL	75.3	91.3	78.4	62.2	67.8	60.6	82.5	75.5	90.4	94.2	96.1	79.5
NNCLR	76.7	93.7	79.0	62.5	67.1	64.1	83.0	75.5	91.8	91.3	95.1	80
SCE	77.7	94.8	80.4	65.3	65.7	59.6	84.0	77.1	90.9	92.7	96.1	80.4
Supervised	72.3	93.6	78.3	61.9	66.7	61.0	82.8	74.9	91.5	94.5	94.7	79.3

SVM classifier on PASCAL VOC 2007/2012

To evaluate the transferability of our pretrained checkpoints on PASCAL VOC by training an SVM classifier, we used the PCL repository.

Method	K = 16	K = 32	K = 64	full
MoCov2	76.14	79.16	81.52	84.60
PCLv2	78.34	80.72	82.67	85.43
ReSSL	79.17	81.96	83.81	86.31
SwAV	78.38	81.86	84.40	87.47
WCL	80.24	82.97	85.01	87.75
SCE	79.47	83.05	85.47	88.24

Object detection and Mask Segmentation on COCO

To evaluate the transferability of our pretrained checkpoints on COCO by training a Mask R-CNN for object detection and mask segmentation, we used the triplet repository.

Method	AP Box	AP Mask
MoCo	40.9	35.5
MoCov2	40.9	35.5
SimCLR	39.6	34.6
BYOL	40.3	35.1
SCE	41.6	36.0
Truncated-Triplet	41.7	36.2
Supervised	40.0	34.7

Acknowledgment

This work was made possible by the use of the Factory-AI supercomputer, financially supported by the Ile-de-France Regional Council.

Licence

CeCILL licence version 2.1.

See LICENSE for details.

Citation

If you found our work useful, please consider citing us:

@InProceedings{Denize_2023_WACV,
    author    = {Denize, Julien and Rabarisoa, Jaonary and Orcesi, Astrid and H\'erault, Romain and Canu, St\'ephane},
    title     = {Similarity Contrastive Estimation for Self-Supervised Soft Contrastive Learning},
    booktitle = {Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
    month     = {January},
    year      = {2023},
    pages     = {2706-2716}
}