DFA-MoE: Tackling Dual Forgetting in Vision-Language Continual Learning

Official PyTorch implementation of the paper "Don't Forget Why You Started: Tackling Dual Forgetting in Vision-Language Continual Learning" ICML2026.

Abstract

Vision-Language Models (VLMs) are strong continual learners, but standard class-incremental learning often damages the pre-trained vision-language alignment that underpins zero-shot generalization. This repository studies that dual-forgetting problem from two perspectives: Incremental Knowledge Forgetting (IKF), which harms previously learned classes, and Pre-trained Knowledge Forgetting (PKF), which erodes the original zero-shot capabilities of the VLM. To address this issue, we implement the Dual-Forgetting-Aware Class-Incremental Learning (DFA-CIL) framework, the Similarity-Calibrated Retention (SCR) metric, and DFA-MoE, a functionally heterogeneous PEFT method that decouples alignment preservation from task adaptation.

Key Contributions

• Dual-forgetting evaluation for VLM continual learning through the DFA-CIL protocol.
• Similarity-Calibrated Retention (SCR) utilizing similarity based weight to disentangle genuine foundational retention from the confounding effects of positive transfer.
• DFA-MoE with two functional pathways:
  • Alignment Pathway: task-agnostic contrastive expert for PKF mitigation.
  • Plasticity Pathway: task-specific experts with classification and auxiliary contrastive learning for IKF mitigation.
  • Hierarchical routing with an inner router over task-specific experts and an outer router balancing alignment and plasticity outputs.

Installation

Setup Environment

git clone https://github.com/RL-MIND/DFA-MoE
cd DFA_MoE
pip install -r requirements.txt

Supported Downstream Datasets

The current configs/class directory provides downstream class-incremental configs for the following dataset indices. These indices are the values used in +train_dataset=[id].

• 0: FGVCAircraft
• 1: Caltech101
• 2: CIFAR100
• 3: DescribableTextures
• 4: EuroSAT
• 5: OxfordFlowers
• 6: Food101
• 7: MNIST
• 8: OxfordPets
• 9: StanfordCars
• 10: SUN397
• 11: Country211
• 14: GTSRB
• 15: RESISC45
• 16: FER2013
• 17: UCF101
• 18: CIFAR10
• 19: STL10
• 20: VOC2007
• 21: ImageNetR
• 22: KittiDistance
• 24: CLEVRCount

The datasets pool also contains:

• 12: SST2
• 13: HatefulMemes
• 23: PCam

These datasets are not supported as --downstream-dataset for calculate_sim.py because they are binary classification datasets.

Data Preparation

We will soon release all the datasets used in this work at google drive.

1. Download or prepare the datasets under a common dataset_root.
2. Update dataset_root in the command line when launching experiments.
3. Choose the dataset-specific config from configs/class/.

Example dataset config files:

• configs/class/eurosat.yaml
• configs/class/flower.yaml
• configs/class/cifar100.yaml
• configs/class/aircraft.yaml

Quick Start

Train a DFA-CIL Run

Example: EuroSAT with 5 class-incremental splits.

python main.py \
  --config-path ./configs/class \
  --config-name eurosat.yaml \
  dataset_root="/path/to/data" \
  +train_dataset=[4] \
  +cil_splits=[5]

Example: OxfordFlowers with 17 splits.

python main.py \
  --config-path ./configs/class \
  --config-name flower.yaml \
  dataset_root="/path/to/data" \
  +train_dataset=[5] \
  +cil_splits=[17]

Important Configuration Options

This project uses Hydra-based configuration. Key parameters include:

model_name: "ViT-B/16"
prompt_template: "a bad photo of a {}."

batch_size: 128
weight_decay: 0.0
ls: 0.0

epochs_a: 1
epochs_b: 1

lr_e1: 1.0e-5
lr_e2: 5.0e-4
text_lr_e2: 1.0e-5
lr_e2_router: 1.0e-3
lr_top_router: 1.0e-5

tau_con: 0.15
tau_b_con: 0.17
lambda_b_con: 0.001

num_task_experts: 2
e2_top_k: 2
moco_queue_size: 128

Pre-task Zero-shot Baseline

To compute SCR correctly, the metric log must contain the original CLIP zero-shot baseline A_k^0. Enable this by setting:

pre_task_zero_shot_eval: true
zero_shot_eval: true

or from the command line:

python main.py \
  --config-path ./configs/class \
  --config-name eurosat.yaml \
  dataset_root="/path/to/data" \
  +train_dataset=[4] \
  +cil_splits=[5] \
  pre_task_zero_shot_eval=true \
  zero_shot_eval=true

This writes a task: -1 entry with zs_pre into metrics.json, which is required by calculate_SCR.py.

SCR Evaluation Workflow

Step 1: Run continual learning and save metrics.json

Run main.py with pre_task_zero_shot_eval=true and zero_shot_eval=true.

Step 2: Generate the similarity matrix

Example: EuroSAT with 5 splits.

python calculate_sim.py \
  --dataset-root "/path/to/data" \
  --downstream-dataset EuroSAT \
  --cil-split 5 \
  --output eurosat_similarity.json

This script:

• uses the original frozen CLIP model,
• produces a task-to-upstream similarity matrix for SCR.

Step 3: Compute SCR

python calculate_SCR.py \
  --metric-path /path/to/metrics.json \
  --sim-json /path/to/eurosat_similarity_sim.json

calculate_SCR.py now strictly requires:

• a task: -1 row with non-empty zs_pre,
• per-task zs results in metrics.json,
• a similarity JSON generated by calculate_sim.py.

Reference Script

The repository also contains run.sh as a reference batch script.

Notes

• main.py, calculate_sim.py, and calculate_SCR.py are the recommended entry points for experiments and evaluation.

Citation

If you find this repository useful in your research, please cite the paper:

@inproceedings{kang2026dont,
  title={Don't Forget Why You Started: Tackling Dual Forgetting in Vision-Language Continual Learning},
  author={Kang, Borui and Gu, Jinrui and Feng, Tao and Fan, Qi and Shi, Yinghuan and Wang, Lei and Li, Wenbin and Gao, Yang},
  booktitle={Proceedings of the 43rd International Conference on Machine Learning},
  year={2026}
}

Acknowledgement

This repository is built upon and modified from MoE-Adapters4CL. We thank the original authors for making their code publicly available.

The dataset processing pipeline in this repository is also based on DIKI. We thank the authors for releasing their implementation.