QuoteBench: Matched Scores Hide Command-Path Failures in LLM Coding Agents

A new standard known as QuoteBench reveals a significant issue in assessing LLM coding agents: matched execution scores may mask command path failures. This benchmark, detailed in arXiv paper 2608.13547, evaluates the transition between command generation and execution transport. It consists of 56 one-shot tasks from 14 incident families, with precise final-state validation. A key feature is an intentionally unescaped parser that intersects the generation contract with execution transport. At the interpolation point, escaping reproduces the raw-path outcomes of each replayed response, indicating that any recovery within a disclosed boundary must arise from changes in the model's generation. In eight same-window configurations, replaying the same response through the added parser results in a success drop of 55.4 to 73.2 percentage points. Disclosure recovers 30.4 to 60.7 points for six configurations, while the other two yield zero or slightly negative results. The raw generation is nearly saturated at the boundary, indicating that while current models excel at raw tasks, they struggle when parsing variations are introduced in the execution environment. This research emphasizes the necessity of assessing the complete command path, rather than solely focusing on the generated text, and offers a technique for distinguishing generation errors from execution failures.