LLM Agent Harness Updating Does Not Predict Performance Gains

A new arXiv preprint (2605.30621) investigates whether a language model's base task-solving capability predicts its ability to self-evolve through harness updates. The study distinguishes two capabilities: harness-updating (producing useful updates from execution evidence) and harness-benefit (benefiting from those updates during task solving). Surprisingly, harness-updating is flat across model capability tiers—models of varying strengths produce updates yielding similar gains. The findings suggest that the ability to generate effective harness updates does not correlate with base performance, and that benefiting from updates is a separate capability. The paper challenges assumptions about self-evolving LLM agents and highlights the need to disentangle evolution capabilities.