LC-GRPO: Langevin Correction Bridges Train-Inference Gap in Flow-Based GRPO
A new framework, LC-GRPO, addresses the train-inference gap in flow-based generative models trained with online reinforcement learning. The method, introduced in arXiv paper 2608.05600, corrects the mismatch between stochastic rollouts used during training and deterministic ODE sampling at inference. By applying a Langevin correction after each ODE Euler step, LC-GRPO aligns training samples with test-time generation, improving policy optimization. The work targets the blurriness caused by exploration noise in SDE rollouts, ensuring consistency in finite-step discretizations.
Key facts
- LC-GRPO is a flow-based GRPO framework with Langevin correction.
- It addresses the train-inference gap in flow-based generative models.
- Online reinforcement learning requires stochastic rollouts, while inference uses deterministic ODE sampling.
- Existing GRPO methods replace ODE with SDE during training, causing sample mismatch.
- SDE rollouts become blurry as exploration noise increases.
- LC-GRPO takes an inference-aligned ODE Euler step followed by a stochastic Langevin correction.
- The method targets the marginal distribution to align training and test-time samples.
- The paper is available on arXiv with ID 2608.05600.
Entities
—