ADAPT: Adversarial Disentangled Prompt Tuning to Combat Robust Overfitting in Vision-Language Models

A novel framework known as ADAPT (Adversarial Disentangled Prompt Tuning) tackles the issue of overfitting in robust generalization within vision-language models. Current adversarial prompt tuning techniques enhance robustness for familiar classes but significantly hinder performance on adversarial samples from unfamiliar classes as training evolves. The authors reveal that this decline is due to the model acquiring pseudo-robust features—shortcuts that lack generalizability. ADAPT employs a dual-prompt strategy, consisting of a target prompt and a set of decoy prompts, adhering to the principle of "Learning What Not to Learn." During training, the decoys capture various pseudo-robust features, while the target prompt is kept orthogonal to them, promoting the development of truly robust features. This disentanglement method seeks to avert shortcut learning and enhance generalization for unseen classes. The paper is accessible on arXiv with the identifier 2608.17306.