SearchMaster: Self-Play Framework for Training LLM Search Agents

A recent paper published on arXiv (2608.01822) presents SearchMaster, a self-play framework designed for training search agents based on large language models (LLMs). This innovative framework allows a single LLM to create, resolve, and validate search tasks within a localized environment, tackling the issue of acquiring high-quality training data. Traditional methods depend on human-created tasks, expert demonstrations, or superior teacher models, whereas SearchMaster seeks to eliminate these constraints by utilizing self-generated tasks. However, self-generated tasks can lead to misleading signals, including pseudo multi-hop questions and inaccurate success-rate estimates. To mitigate these issues, SearchMaster implements three controls: an Evidence-Chain Generator (ECG), a Search-Depth Reward (SDR), and a third, less detailed control likely aimed at enhancing rollout quality. The paper details these approaches to advance training for search agents.