NeurIPS 2026

Decoupled Mode Connectivity for Base-to-Novel Generalization in Vision-Language Models

Imad Eddine Marouf1 Khalid Oublal2 Enzo Tartaglione1 Stéphane Lathuilière3

1Télécom Paris, Institut Polytechnique de Paris 2Google DeepMind 3Inria, Grenoble, France

Prompt
Prompts between the zero-shot goal and the base-class goal zero-shot features base-class optimum cross-entropy anchor R c cgen cspec α = 0.20
Accuracy schematic
Base versus novel accuracy: one prompt slides along a single trade-off curve, the DMC corridor traces a curve beyond it higher HM ↗ base accuracy → novel accuracy → one prompt, every λ DMC corridor, every α base ↑ novel ↑ c

Prompt learning on CLIP trades accuracy on unseen classes for accuracy on the training classes, because a single prompt has to follow two gradients that point in nearly opposite directions. DMC gives each objective its own prompt, couples the two through a low-loss linear corridor in text-feature space, and deploys one point on that corridor, at the inference cost of a single prompt.

The trade-off

We tune CLIP's text prompt on the base half of each dataset's classes and test on both halves. Averaged over 11 datasets, CoOp gains 13.4 points on base classes and loses 11.0 on novel ones. The harmonic mean ends where zero-shot CLIP started.

zero-shot CLIPCoOp
Base69.3
Novel74.2
HM71.7
050100
Average accuracy over 11 datasets (%). Tick marks show zero-shot CLIP.

Why a better loss does not fix it

Methods such as KgCoOp and MergeTune add a regularizer \(\mathcal{R}\) that pulls the prompt back toward CLIP's zero-shot text features. When one prompt \(\mathbf{c}\) carries both terms, every stationary point of \(\mathcal{L}_{\mathrm{CE}} + \lambda\mathcal{R}\) satisfies

\[ \htmlClass{m-spec}{\nabla_{\mathbf{c}}\,\mathcal{L}_{\mathrm{CE}}} \;=\; -\lambda\,\htmlClass{m-gen}{\nabla_{\mathbf{c}}\,\mathcal{R}}. \]

No step from there improves both objectives, and changing \(\lambda\) slides the solution along the trade-off rather than past it. The opposition shows up in practice: at the converged single-prompt model, the cosine \(\gamma\) between the two gradients is negative on every dataset and baseline we measured.

CoOpKgCoOp
−1−0.50
\(\gamma = \cos(\nabla_{\mathbf{c}}\mathcal{L}_{\mathrm{CE}},\,\nabla_{\mathbf{c}}\mathcal{R})\) at the converged MergeTune prompt, over the full training set.

Swapping the regularizer does not help either. Seven alternatives on the same single prompt, among them Fisher weighting, prompt-space EWC and zero-shot distillation, gained at most 0.40 HM on Caltech101 and Flowers102, about what retuning \(\lambda\) gives.

Two prompts and a corridor

DMC starts two prompts from the same fine-tuned checkpoint. The specialization prompt \(\mathbf{c}_{\mathrm{spec}}\) is trained by cross-entropy, the generalization prompt \(\mathbf{c}_{\mathrm{gen}}\) by the anchor. The only term that couples them is a corridor: every normalized mixture of their text features must classify base images well.

\[ \htmlClass{m-corr}{\widetilde f(\alpha)} = \ell_2\big(\htmlClass{m-gen}{f_{\mathrm{gen}}} + \alpha\,(\htmlClass{m-spec}{f_{\mathrm{spec}}} - \htmlClass{m-gen}{f_{\mathrm{gen}}})\big) \]
\[ \mathcal{L}_{\mathrm{DMC}} = \underbrace{\htmlClass{m-spec}{\mathcal{L}_{\mathrm{CE}}(f_{\mathrm{spec}}) + \lambda_{\mathrm{VA}}\mathcal{L}_{\mathrm{VA}}}}_{\htmlClass{m-spec}{\mathbf{c}_{\mathrm{spec}}\text{ only}}} + \underbrace{\htmlClass{m-gen}{\lambda_{\mathrm{gen}}\,\mathcal{R}(f_{\mathrm{gen}}, f_{w_1})}}_{\htmlClass{m-gen}{\mathbf{c}_{\mathrm{gen}}\text{ only}}} + \underbrace{\htmlClass{m-corr}{\beta\,\mathbb{E}_{\alpha\sim\mathcal{U}[0,1]}\,\mathcal{L}_{\mathrm{CE}}\big(\widetilde f(\alpha)\big)}}_{\htmlClass{m-corr}{\text{corridor, both}}} \]
\[ \begin{aligned} \mathcal{L}_{\mathrm{DMC}} = {} & \htmlClass{m-spec}{\mathcal{L}_{\mathrm{CE}}(f_{\mathrm{spec}}) + \lambda_{\mathrm{VA}}\mathcal{L}_{\mathrm{VA}}} \\ {}+{} & \htmlClass{m-gen}{\lambda_{\mathrm{gen}}\,\mathcal{R}(f_{\mathrm{gen}}, f_{w_1})} \\ {}+{} & \htmlClass{m-corr}{\beta\,\mathbb{E}_{\alpha}\,\mathcal{L}_{\mathrm{CE}}\big(\widetilde f(\alpha)\big)} \end{aligned} \]

The corridor is drawn in text-feature space because logits are linear in the text features, so every point of the segment is itself a classifier. A segment between prompts would not have this property. After training we fix \(\alpha = 0.20\) for all datasets and merge the features once.

The corridor from f_gen to f_spec; the ring marks the classifier selected by alpha and the star marks alpha = 0.20. fgen fspec
0.20

Visual Anchor

Freed from the anchor, cross-entropy tends to push base classes apart and erase the smaller similarities CLIP keeps between related classes. The Visual Anchor matches CLIP's zero-shot class-similarity distribution on each training image with a KL term that never reads the label. It is a refinement: in our ablation it helps on two of four datasets.

Class-permutation invariance

A regularizer that depends on which class is which leaks base-class identity into the prompt. Under idealized assumptions, Fano's inequality turns that leak into a lower bound on the drop in novel accuracy. We rely on the bound for the direction of the effect, not its size. Both regularizers in DMC are invariant.

Results

Across 11 datasets, DMC raises the average HM of MergeTune, the single-prompt method it builds on, from 76.45 to 77.23 with CoOp and from 76.43 to 76.76 with KgCoOp.

Three loss-landscape panels on Flowers102. Left: the straight path from zero-shot CLIP to the single-prompt baseline crosses a high-loss region. Middle: at epochs 0, 5, 50 and 100 the two DMC endpoints start together and move apart. Right: the trained corridor lies in a low-loss valley, with the operating point marked by a star; an inset shows base, novel and harmonic-mean accuracy along alpha.
Joint base and novel cross-entropy on Flowers102, in a 2-D slice of text-feature space. The path from zero-shot CLIP to the single-prompt baseline crosses high loss. The two DMC prompts start together, move apart, and end with a corridor in a valley that path never enters.
Base versus novel accuracy on Caltech101, SUN397, UCF101 and EuroSAT. Each DMC curve sweeps alpha from the generalization to the specialization endpoint; the operating point at alpha 0.20 lies above and to the right of the MergeTune point.
Each curve sweeps \(\alpha\) from \(\mathbf{c}_{\mathrm{gen}}\) to \(\mathbf{c}_{\mathrm{spec}}\). On these four datasets the star (\(\alpha = 0.20\)) sits up and to the right of MergeTune (gray): base and novel accuracy rise together.
HM of DMC minus HM of MergeTune. DMC is ahead on 8 of the 10 non-ImageNet datasets with CoOp and 7 of 10 with KgCoOp; the drops are within seed spread. MergeTune and DMC results on this page are means over three seeds at \(\alpha = 0.20\).

Is it the extra prompt?

DMC trains two prompts of 2K parameters each. A single CoOp prompt with 8K parameters still trails it on all four datasets we tested, by 1.1 to 5.1 HM. Two prompts trained without the anchor score below MergeTune, because both collapse onto the same base-specialized solution.

1 prompt, 2K4K8KDMC, 2 × 2K
HM with CoOp as the base method.

Where it does not help

DMC needs its two endpoints to stay apart after training. On the prompt-tuning baselines they do. On MMA, whose adapter is small, both endpoints collapse onto the same classifier and the corridor shrinks to a point. MMA with DMC still improves on MMA with MergeTune, but both trail the published MMA result. Separation is necessary, not sufficient.

CoOp0.600
KgCoOp0.719
MMA0.993
00.51
\(\cos(f_{\mathrm{gen}}, f_{\mathrm{spec}})\) after training, averaged over 10 datasets. Near 1, the two endpoints are the same classifier.

The operating point is also chosen offline. The best \(\alpha\) differs across datasets and cannot be read from base classes alone, which leaves a small gap to the best point on each corridor.

Citation

@inproceedings{marouf2026dmc,
  title     = {Decoupled Mode Connectivity for Base-to-Novel
               Generalization in Vision-Language Models},
  author    = {Marouf, Imad Eddine and Oublal, Khalid and
               Tartaglione, Enzo and Lathuili{\`e}re, St{\'e}phane},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026}
}