Self-Guided Process Reward Optimization for LLM Reasoning

A new framework called Self-Guided Process Reward Optimization (SPRO) has been proposed to enhance the reasoning capabilities of Large Language Models (LLMs) through process reinforcement learning. SPRO eliminates the need for additional process reward models by deriving process rewards intrinsically from the policy model itself. It introduces two key innovations: Cumulative Process Rewards (CPR) and Masked Step Advantage (MSA), which enable rigorous step-wise action advantage estimation within shared-prompt sampling groups. The approach addresses the lack of a unified theoretical framework for process-level advantage estimation and reduces computational overhead. Experimental results demonstrate its effectiveness in improving LLM reasoning.