Echo Gap: Memory Reward Inflation in Self-Improving LLM Agents

A recent study published on arXiv (2608.00017) examines a specific failure mode in self-improving large language model (LLM) agents that enhance their performance through experience without updating weights. These agents utilize external memory to store episodes, evaluate them, and later retrieve similar ones for future tasks. The score assigned to each episode serves as a proxy reward for a non-parametric policy, with each retrieval functioning as a step toward policy improvement. However, during deployment, the absence of ground-truth labels means that the stored rewards reflect only the LLM's evaluations. This leads to the 'Echo Gap' phenomenon in memory-based self-improving agents, where incorrect episodes receive exaggerated rewards, prompting the agent to repeat its most confident errors. The paper is noted as a new submission and can be accessed via the provided URL.