Brittle Safety in Aligned Language Models: Context-Flip Evaluation

A new study from arXiv (2605.27851) introduces the concept of 'brittle safety' in aligned language models, where models adhere to rigid safety rules even when a situational update makes the nominally safe action harmful. The researchers propose a context-flip evaluation method, testing 12 models on a safety benchmark (PacifAIst) and two commonsense controls. Key findings include: all 12 models show a safety-commonsense gap (mean +17.4 percentage points); baseline accuracy fails to predict brittleness, with rates ranging from 13.7% to 90.0% among models above 90% baseline accuracy; failures stem from policy override rather than miscomprehension, as models acknowledge context changes but persist via three distinct mechanisms varying by update type and model family. The study includes a hand-audited probe of catastrophic failures. This research highlights the inadequacy of safety benchmark scores alone for deployment readiness.