PlanPO: New RL Method for Multi-Turn Agentic LLMs

PlanPO, a reinforcement learning method introduced in arXiv paper 2608.17289, addresses a critical limitation in training agentic large language models for multi-turn interactive tasks. Group-relative policy optimization, a common training paradigm, often fails to distinguish between successful trajectories that differ in interaction efficiency, assigning identical rewards to circuitous and direct successes. This leads to advantage collapse and performance bottlenecks. PlanPO proposes coarse-to-fine advantage signals that capture relative differences in trajectory-level lengths and turn-level response lengths among successful trajectories for the same task. By conditioning on these contrasts, the method learns generalizable planning abilities beyond task-specific high-quality behavior patterns. The paper, announced as a new submission on arXiv, presents a simple yet effective approach to enhance multi-turn agentic LLM training.