CAST: Using Game Solvers to Guide LLM Agent Training

A novel approach known as CAST (Credit Assignment from Solver Teachers) enhances the training process of large language models (LLMs) for extended games by utilizing game solvers to deliver feedback at each turn. Traditional reinforcement learning with verifiable rewards (RLVR) offers only limited final rewards, complicating the task of attributing success to specific actions. CAST translates variations in a solver's state value into solver advantages, integrating these as dense signals within RLVR. Assuming a soft-optimal solver, maximizing solver advantage aligns with on-policy distillation from the solver, relying solely on scalar values. This technique was evaluated using games such as Sokoban and Minesweeper, among others.