Feedback-Aware Credit Assignment Improves Multi-Turn Agent Training
A new paper available on arXiv under identifier 2608.17499 presents a novel approach called Feedback-Aware Credit Assignment (FACA), designed to enhance multi-turn agent training. FACA utilizes the next user turn as a source of noisy, temporally local evidence regarding the preceding user-to-user interaction. Unlike traditional interactive reinforcement learning, which relies on a single terminal reward per rollout, FACA integrates a locally normalized reaction advantage with a validated terminal outcome advantage. Experimental results showed that FACA improved the nine-domain τ-family average by 5.91 and 10.22 percentage points across three separate training runs, compared to a baseline Interactive GRPO control.
Key facts
- The paper is available on arXiv under identifier 2608.17499.
- The paper introduces a method called Feedback-Aware Credit Assignment (FACA).
- FACA leverages the next user turn as noisy, temporally local evidence about the preceding user-to-user segment.
- Standard interactive reinforcement learning typically uses a single terminal reward for each rollout.
- FACA combines a locally normalized reaction advantage with a verified terminal outcome advantage.
- FACA does not require an extra critic or rollout.
- In experiments, FACA improved the nine-domain τ-family average by 5.91 and 10.22 percentage points across three independently trained runs.
- The baseline used was an outcome-only Interactive GRPO control matched in simulator, visible dialogue, initialization, rollout, and optimization.
Entities
—