CrEST: Hierarchical Credit Assignment for Multi-Turn LLM Agents

The introduction of a new framework called CrEST aims to enhance credit assignment in reinforcement learning with verifiable rewards (RLVR) for training agents that utilize tools over multiple turns. This approach tackles the drawbacks of trajectory-level credit assignment, which merges diverse outcomes from each turn into one reward signal, as well as the issues with on-policy distillation, which is limited by teacher constraints or susceptible to gradient concentration collapse. CrEST maintains the verifier-bounded ceiling of RL while integrating dense token-level signals from a self-teacher. It addresses credit assignment on two fronts: verified advantages segmented by turn mitigate inter-turn dilution, and entropy-gated modulation from the self-teacher improves intra-turn token contributions. Experiments conducted on BFCL V3 and WildToolBench demonstrate that CrEST consistently surpasses both RL and distillation baselines across two model scales. The paper is accessible on arXiv (2608.13179).