LLM Orchestration Causes Universal Defect Detection Cliff
A study on arXiv (2605.26174) reveals that when production language-model systems use orchestrated worker agents to answer requests, they universally lose the ability to detect cross-section defects—contradictions between distant document sections. Testing ten models across five generations from one developer and five providers with distinct alignment paradigms, researchers found a universal detection cliff: every model that could find these defects under a single agent failed under orchestration, with detection dropping two-thirds or more. The cliff is mechanism-derived and not mitigated by scale or extended reasoning. Among the six models discriminating above chance after the fall, a signal-detection decomposition showed varied behavior.
Key facts
- arXiv paper 2605.26174
- Cross-section defect detection studied
- Ten models tested across five generations and five providers
- Universal detection cliff observed under orchestration
- Detection drops two-thirds or more
- Cliff is mechanism-derived, not closed by scale or reasoning
- Six models discriminated above chance after the fall
- Signal-detection decomposition applied
Entities
Institutions
- arXiv