FP4 Reinforcement Learning Post-Training for Large Language Models

Researchers present the first end-to-end FP4 reinforcement learning post-training for large language models, where both rollout and training policies operate at 4-bit precision. A systematic study identifies rollout activation quantization as the primary source of degradation, not training-side quantization. Outliers in activations cause underflow to zero under FP4. Restoring training policy to higher precision while keeping rollout in FP4 worsens accuracy, revealing rollout-training mismatch as the main failure mode. To address this, they propose Rollout Residual Quantization (Rollout-ResQ), a single residual correction term with a hardware-friendly sparsity pattern added only to the FP4 rollout matrix multiplication. This lightweight correction recovers performance.