Fundamental flaw makes LLMs impossible to fully secure, researchers warn

A team of researchers, including Charles Ye and Jasmine Cui, presented a paper at the International Conference on Machine Learning arguing that large language models (LLMs) have a fundamental vulnerability that cannot be fixed through training or red-teaming. The flaw lies in how LLMs identify roles: they infer who is giving instructions based on text style rather than structural tags. By mimicking the style of a model's own chain-of-thought, attackers can spoof roles and bypass guardrails. The researchers demonstrated attacks on models from OpenAI, Anthropic, Alibaba, and DeepSeek, extracting instructions for synthesizing cocaine and sabotaging aircraft navigation systems. The attack, called chain-of-thought forgery, won OpenAI's red-teaming hackathon in August 2025. The researchers advise assuming LLMs cannot be trusted and treating all AI agent actions as potentially unsafe. Florian Tramèr of ETH Zürich praised the paper but noted that current defenses work well for less sensitive cases. Cui and Ye warn that economic incentives will drive more attacks, and no fundamental solution exists.