Step-Level Self-Distilled Policy Optimization for Deep Search Agents

A recent study published on arXiv (2608.12764) presents Step-Level Self-Distilled Policy Optimization (SDS-PO), aimed at enhancing deep search agents through the provision of dense, step-level rewards. Traditional reinforcement learning typically delivers a single outcome reward for each trajectory, which proves insufficient for effective credit assignment in multi-step search scenarios. On-policy self-distillation (OPSD) leverages the model's own logits as dense token-level teachers; however, applying this to search agents introduces a conflict since the teacher possesses privileged information (such as the correct answer), resulting in a distribution that diverges from the student's exploration-based reasoning. To address this, the authors introduce Evidence Anchors—brief, step-level evidence snippets sourced from the web—serving as privileged information that encapsulates essential reasoning steps without disclosing the complete answer. This approach enables the student to develop improved search strategies without adopting the teacher's informational advantage. The paper also serves as a cross-announcement on arXiv, suggesting it may have been previously presented elsewhere. This method tackles a critical issue in training deep search agents, which navigate trajectories that can extend over numerous steps, making it significant for AI research, especially in reinforcement learning and search-based reasoning.