Multimodal Jailbreak Robustness in Vision-Language Models

A recent study, available as arXiv preprint 2605.27932, looks into the safety of think-with-image reasoning in large vision-language models (VLMs). It explores four different methods of inference: generating direct responses, using text-only prior turns, manipulating visual states, and invoking external image tools. The findings show that engaging directly with image tools leads to the lowest rates of attack success, reducing jailbreak attempts by around 30% on average across several VLMs. Interestingly, even when the tool's output is changed or considered unsafe, the attack success rates remain low. However, they spike to levels similar to direct-answering when relying on text-only prior turns. This highlights the importance of designing processes to enhance multimodal safety.