New Research Proposes Hierarchical Framework to Strengthen Vision-Language Models Against Adversarial Attacks

A new framework for adversarial fine-tuning has been developed to improve the resilience of Vision-Language Models (VLMs) against targeted threats. This method utilizes the natural hierarchical organization of class spaces, tackling a weakness where models falter when adversarial attacks focus on both superclasses, such as 'mammal', and their specific leaf classes like 'cat'. Current robust fine-tuning strategies often match fixed text embeddings with image embeddings, potentially undermining overall performance and strength. The innovative framework introduces hierarchical embeddings and establishes multiple levels of adversarially robust alignment between the text and image modalities. It also incorporates mechanisms to place visual embeddings at specific depths within the hierarchy. A theoretical link between the embedding depth in the hierarchy and the maximum feasible margin size is established. This research, presented in the paper 'Hierarchically Robust Zero-shot Vision-language Models' (arXiv:2604.18867v1), represents a cross-disciplinary effort aimed at enhancing zero-shot classification in VLMs, which remain vulnerable to adversarial attacks despite their sophisticated capabilities.