Vision Transformers Exhibit Benign Overfitting in Adversarial Training, New Research Shows
A recent theoretical study indicates that Vision Transformers (ViTs) can attain effective robust generalization through adversarial training, even in instances of benign overfitting. This research, identified as arXiv:2604.19724v1, marks the inaugural theoretical exploration of adversarial training within simplified ViT frameworks. Although ViTs have shown impressive performance in numerous vision tasks, they are still susceptible to adversarial examples, akin to Convolutional Neural Networks (CNNs). The findings suggest that under certain conditions—such as a moderate perturbation budget and a specific signal-to-noise ratio—ViTs can reach nearly zero robust training loss and robust generalization error in specific scenarios. This discovery broadens the understanding of benign overfitting, previously noted only in CNNs, to include Vision Transformers.
Key facts
- Vision Transformers (ViTs) remain vulnerable to adversarial examples despite their success in vision tasks
- Adversarial training is a common empirical defense strategy for ViTs
- Theoretical underpinnings of adversarial training robustness in ViTs were previously unexplored
- This work presents the first theoretical analysis of adversarial training under simplified ViT architectures
- ViTs can achieve nearly zero robust training loss and robust generalization error under specific conditions
- Benign overfitting—strong generalization despite overfitting—occurs in ViTs with adversarial training
- Benign overfitting was previously observed only in Convolutional Neural Networks (CNNs) with adversarial training
- The research was published as arXiv:2604.19724v1 with announcement type: cross
Entities
—