RISE-RL: New Method Improves Open-Ended RL for LLMs

A recent study introduces RISE-RL (Rubric-Informed Selective Exploration), a novel approach aimed at enhancing reinforcement learning (RL) for the alignment of Large Language Models (LLMs) in open-ended tasks. These tasks present the difficulty of requiring responses that meet various criteria, with no single correct answer. Traditional rubric-based RL techniques condense detailed feedback into scalar rewards, complicating the identification of ongoing capability deficiencies during limited on-policy exploration. RISE-RL addresses this by focusing on frequently overlooked rubric criteria to uncover valuable trajectories that standard exploration might miss. It keeps only those trajectories where the complete-rubric reward surpasses the average reward from natural rollouts and then reassesses them based on the original prompt to highlight behaviors that the natural policy inadequately supports. The guidance signal is refined through a distinct auxiliary objective and is discarded post-training. The research is accessible on arXiv under ID 2608.09123.