Selective Importance Sampling Method Aims to Improve LLM Alignment in RL Post-Training

A recent preprint on arXiv, identified as number 2607.04728, introduces Selective Importance Sampling (SIS), a method designed to improve alignment in large language models during the reinforcement learning (RL) post-training phase. This phase typically employs a 'rollout then update' strategy, which leads to the creation of off-policy training data. While importance sampling (IS) aims to rectify this distribution mismatch, the token-level importance ratios can lead to significant variance in long sequences. SIS, drawing from rejection sampling, resolves this by converting off-policy tokens into on-policy tokens, eliminating the need for importance score corrections for accepted tokens. The authors assert that SIS narrows the gap between token-level and sequence-level training, presenting a valuable approach for enhancing LLM alignment. The paper can be found at https://arxiv.org/abs/2607.04728.