CoRT: Token-Level Credit Assignment for Rubric-Guided RL in LLMs
A novel approach known as CoRT (Counterfactual Replay for Token-Level Rubric-Guided Policy Optimization) tackles a challenge in rubric-based reinforcement learning for language models. Traditional GRPO-style frameworks simplify rubric assessments into a single scalar reward, which is uniformly applied to all tokens, resulting in a lack of specific credit distribution. CoRT, however, implements token-level credit weighting without the need for an additional model. It employs counterfactual replay by rescoring the same sampled response using both the original rubric-conditioned prompt and a corresponding criteria-free prompt. The contrasts in tokenwise log-likelihood serve as a proxy for rubric context dependence, which are then converted into bounded, response-normalized weights. This method is elaborated in a paper available on arXiv (2607.25659).
Key facts
- CoRT is a token-level credit weighting method for rubric-conditioned GRPO.
- It uses counterfactual replay to rescore responses under rubric and criteria-free prompts.
- Tokenwise log-likelihood contrasts proxy rubric context dependence.
- No auxiliary token scoring model is required.
- Standard GRPO pipelines broadcast scalar rewards uniformly to all tokens.
- The paper is available on arXiv with ID 2607.25659.
- Rubric-based RL evaluates outputs against explicit criteria.
- CoRT maps contrasts to bounded, response-normalized weights.
Entities
Institutions
- arXiv