MOS: Model Surgery for Pre-Trained Model-Based Class-Incremental Learning

Hai-Long Sun^1,2  Da-Wei Zhou^1,2  Hanbin Zhao³  Le Gan^1,2  De-Chuan Zhan^1,2  Han-Jia Ye^1,2 

¹School of Artificial Intelligence, Nanjing University  ²State Key Laboratory for Novel Software Technology, Nanjing University  ³College of Computer Science and Technology, Zhejiang University 

🎉The code repository for "MOS: Model Surgery for Pre-Trained Model-Based Class-Incremental Learning" (AAAI 2025) in PyTorch. If you use any content of this repo for your work, please cite the following bib entry:

  @inproceedings{sun2024mos,
    title={MOS: Model Surgery for Pre-Trained Model-Based Class-Incremental Learning},
    author={Sun, Hai-Long and Zhou, Da-Wei and Zhao, Hanbin and Gan, Le and Zhan, De-Chuan and Ye, Han-Jia},
    booktitle={AAAI},
    year={2025}
  }

Overview

Class-Incremental Learning (CIL) requires models to continually acquire knowledge of new classes without forgetting old ones. Despite Pre-trained Models (PTMs) have shown excellent performance in CIL, catastrophic forgetting still occurs as the model learns new concepts. Existing work seeks to utilize lightweight components to adjust the PTM, while the forgetting phenomenon still comes from parameter and retrieval levels. Specifically, iterative updates of the model result in parameter drift, while mistakenly retrieving irrelevant modules leads to the mismatch during inference.

To this end, we propose MOdel Surgery (MOS) to rescue the model from forgetting previous knowledge. By training task-specific adapters, we continually adjust the PTM to downstream tasks. To mitigate parameter-level forgetting, we present an adapter merging approach to learn task-specific adapters, which aims to bridge the gap between different components while reserve task-specific information. Besides, to address retrieval-level forgetting, we introduce a training-free self-refined adapter retrieval mechanism during inference, which leverages the model's inherent ability for better adapter retrieval. By jointly rectifying the model with those steps, MOS can robustly resist catastrophic forgetting in the learning process. Extensive experiments on seven benchmark datasets validate MOS's state-of-the-art performance.

🎊 Results

We conducted experiments on seven benchmark datasets to verify the competitive performance of MOS.

Requirements

🗂️ Environment

1. torch 2.0.1
2. torchvision 0.15.2
3. timm 0.6.12

🔎 Dataset

We provide the processed datasets as follows:

• CIFAR100: will be automatically downloaded by the code.
• CUB200: Google Drive: link or Onedrive: link
• ImageNet-R: Google Drive: link or Onedrive: link
• ImageNet-A: Google Drive: link or Onedrive: link
• OmniBenchmark: Google Drive: link or Onedrive: link
• VTAB: Google Drive: link or Onedrive: link
• ObjectNet: Onedrive: link You can also refer to the filelist if the file is too large to download.

You need to modify the path of the datasets in ./utils/data.py according to your own path.

These datasets are referenced in the Aper

🔑 Running scripts

Please follow the settings in the exps folder to prepare json files, and then run:

python main.py --config ./exps/[filename].json

Here is an example of how to run the code

if you want to run the cifar dataset using ViT-B/16-IN1K, you can follow the script:

python main.py --config ./exps/mos_cifar.json

if you want to run the cifar dataset using ViT-B/16-IN21K, you can follow the script:

python main.py --config ./exps/mos_cifar_in21k.json

After running the code, you will get a log file in the logs/mos/cifar224/ folder.

👨‍🏫 Acknowledgment

We would like to express our gratitude to the following repositories for offering valuable components and functions that contributed to our work.

• PILOT: A Pre-Trained Model-Based Continual Learning Toolbox
• RevisitingCIL