LEGO-RL: New Framework Aligns Coding-Agent Harnesses with Policy-Gradient Training
The integration of reinforcement learning for coding agents depends on extensive agent harnesses for tool execution and feedback. Unfortunately, these harnesses often conflict with policy-gradient training, resulting in environmental failures and reward manipulation. To tackle these challenges, LEGO-RL links coding-agent harnesses with scalable policy-gradient optimization while maintaining their control flow. It is founded on three essential components: accurate optimization via in-process LLM proxying for token-level alignment, and dependable execution through scalable sandbox orchestration. The goal of this framework is to minimize inconsistencies in RL training by keeping the harness's original control flow intact. This research is outlined in a preprint on arXiv under identifier 2608.17393, aiming to improve the reliability of RL training for coding agents.
Key facts
- LEGO-RL is a framework for reinforcement learning in coding agents.
- It targets long-running agent harnesses for tool integration, repository context, and execution feedback.
- Native execution environments are misaligned with policy-gradient training.
- Environmental crashes and reward hacking corrupt outcome signals.
- Train-inference discrepancies decouple rollout behavior from policy updates.
- LEGO-RL bridges harnesses with policy-gradient optimization without modifying internal control flow.
- The framework uses in-process LLM proxying to capture raw generation streams for token-level alignment.
- The paper is available on arXiv as 2608.17393.
Entities
—