OPPO: Bayesian Token-Level Credit Assignment for LLM Reasoning

A new reinforcement learning method for large language models, Oracle-Prompted Policy Optimization (OPPO), addresses the credit assignment problem in token-level reasoning. Unlike GRPO, which assigns a single trajectory-level advantage to all tokens, OPPO uses a Bayesian update of the model's belief about eventual success to provide per-token signals. This approach accumulates oracle signals along a trajectory to estimate success probability at each position, requiring only one extra forward pass. The method improves upon prior distillation-style techniques by integrating local discrimination with trajectory-level evidence.