FlowLM: Few-Step Text Generation via Diffusion-to-Flow Adaptation
FlowLM is a language model that utilizes flow matching, developed from pre-trained diffusion language models through a streamlined fine-tuning process. By transforming the curved sampling paths of diffusion models into linear flows, FlowLM enables high-quality generation in just a few steps, achieving results that either match or exceed those of 2,000-step diffusion sampling while requiring minimal training epochs. The fine-tuned version of FlowLM reaches its peak performance with only half the training epochs compared to starting from scratch, with both methods significantly outperforming the original diffusion model. Additionally, the research confirms a more effective training goal for flow matching: predicting clean data to consistently steer the sampling towards the actual data distribution. Empirical findings highlight the method's effectiveness in producing high-quality, few-step text generation.
Key facts
- FlowLM is a flow matching language model.
- It is transformed from pre-trained diffusion language models via efficient fine-tuning.
- It re-aligns curved sampling trajectories of diffusion models into straight-line flows.
- It enables high-quality few-step generation rivaling 2,000-step diffusion sampling.
- Fine-tuned FlowLM reaches performance saturation with half the training epochs of training from scratch.
- Both fine-tuned and scratch-trained models outperform the original diffusion model.
- A more effective training objective is predicting clean data.
- The approach is validated for high-quality, few-step text generation.
Entities
Institutions
- arXiv