Hallucination-to-Action: New Attack Surface in Multimodal Agents

A recent study published on arXiv (2605.19192) addresses a significant security flaw in multimodal AI agents, specifically the conversion of hallucinations into actions. When a misleading visual assertion leads to an important action, such as a click, email, or financial transaction, it results in an authorization error instead of simply a quality issue. To counter this, the authors introduce evidence-carrying multimodal agents (ECA). This approach considers free-form model text as inadmissible, breaks down each tool invocation into essential predicates, and acquires typed certificates from restricted DOM, OCR, and accessibility checks. A deterministic gate is employed to ensure only authorized privileges are granted. The architecture transforms unclear model beliefs into identifiable verifiers, schemas, and implementation residues. Testing against over 1,900 attacks reveals these residues. The paper falls under cs.CR, cs.AI, and cs.LG categories.