Coding Agent Study Quantifies Coherence Debt Across Seven Models
An arXiv preprint (2608.16630) presents the idea of coherence debt in repository-scale coding, where agents are tasked with maintaining consistency among tests, imports, configurations, and migration rules within a defined context window. The authors conceptualize this as the reconstruction of a coupled-fact graph, where each modification necessitates facts from recent context or parametric memory; debts arise from facts not covered by either. In their experiments with seven models and five harnesses, they manipulated these channels and introduced faults. No model succeeded on an unseen API when both channels were empty, but success returned when facts were included in the prompt. A rename that disrupted memorized knowledge of a real library led to identical failures across all seven models, which passed and failed the same tests. The findings indicate that availability, rather than distance, influences outcomes: withholding a fact incurs a cost equal to the work it supports, and a provided fact is equally effective regardless of proximity to the edit. Harnesses incur different costs, as configurations that pass all tests do not uniformly withstand missing facts.
Key facts
- The paper is titled 'The Working Set of a Coding Agent: Coherence Debt in Repository-Scale Tasks' and posted on arXiv with identifier 2608.16630.
- Repository-scale coding requires maintaining consistency of tests, imports, configuration, and migration rules within a bounded context window.
- The study models coding agent behavior as reconstructing a coupled-fact graph.
- Required facts originate from either recent context or parametric memory; facts covered by neither create coherence debt.
- Experiments involved seven models and five harnesses, with channels supplied or withheld and faults injected.
- No model succeeded on an unseen API when both channels were empty; adding facts to the prompt restored success.
- When a rename contradicted a real library's memorized details, all seven models failed in the same place, passing and missing identical tests.
- Availability of facts determines performance, not proximity to the edit: withholding a fact costs exactly the work it supports, while supplied facts work equally well regardless of distance.
- Different harnesses incur unequal costs from missing facts.
- The source is an arXiv abstract (announce type: cross).
Entities
—