EnterpriseRAG Benchmark Exposes LLM Instruction Adherence Collapse in Production

The newly introduced benchmark, EnterpriseRAG, highlights a significant reliability issue in enterprise retrieval-augmented generation (RAG) implementations. Although large language models (LLMs) fulfill 80% of individual criteria, only 26.8% of their outputs satisfy all conditions at once, revealing a 57-point orchestration shortfall. Current benchmarks operate under the assumption of straightforward queries with clean retrieval, neglecting the complexities of real-world scenarios that involve noisy documents and multi-faceted constraints. Documented in a paper on arXiv (2608.11584), EnterpriseRAG features 983 expert-validated examples across six domains, effectively simulating three failure modes overlooked in previous studies: retrieval noise, knowledge deficiencies, and factual discrepancies, alongside intricate instructions. An assessment of 13 leading LLMs uncovers a significant drop in adherence to instructions, where high satisfaction of individual constraints obscures poor overall compliance, revealing substantial obstacles related to knowledge gaps and factual inconsistencies, even with enhanced reasoning capabilities, indicating the need for further improvements in production RAG.