Prefix-RFT: Hybrid LLM Post-Training Method

A new hybrid approach to large language model post-training, Prefix-RFT, combines supervised fine-tuning (SFT) and reinforcement fine-tuning (RFT) to overcome their respective limitations. SFT excels at mimicking demonstration data but suffers from behavior cloning, while RFT enhances performance but is sensitive to initial policy and prone to unexpected behaviors. Prefix-RFT synergizes learning from both demonstration and exploration, using mathematical reasoning problems as a test bed. The method outperforms standalone SFT, standalone RFT, and parallel mixed-policy RFT. The paper highlights the complementary nature of SFT and RFT, proposing a unified view of these techniques.