AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning

A recent paper published on arXiv (2608.05987) presents AgentOPSD, a novel recursive technique for turn-level credit assignment in agentic reinforcement learning that does not require a critic. This approach tackles a significant challenge in reinforcement learning involving verifiable rewards: trajectory-level advantage assessments often neglect the critical decisions that influence outcomes in lengthy, multi-turn tasks. AgentOPSD compiles token-level teacher-student log-probability discrepancies into turn-level evidence and updates a Bayesian belief state in log-odds space recursively, creating a systematic reweighting method that transforms sparse outcome supervision into turn-level credit indicators. The identification of crucial turns is achieved through marginal belief adjustments between successive states. This method is fully compatible with traditional policy gradient techniques. The authors of the paper are not specified in the abstract.