DiffusionGemma: A New Open-Weight Model for High-Speed Text Generation
Researchers have unveiled DiffusionGemma, an innovative open-weight language model that utilizes discrete diffusion for rapid text generation. In contrast to traditional autoregressive (AR) large language models that process one token sequentially, DiffusionGemma refines blocks of 256 tokens simultaneously, eliminating the delays associated with sequential decoding. This model is fine-tuned from the Gemma 4 mixture-of-experts architecture, which consists of 3.8 billion active parameters out of a total of 25.2 billion. Its training is notably efficient, employing a two-stage approach that uses less than 10% of the AR model's total training token budget. The first stage focuses on supervised fine-tuning for bidirectional denoising, while the second integrates reinforcement learning with sampler distillation to enhance both generation quality and inference speed. DiffusionGemma sets a new standard for balancing generation speed and quality, making it a valuable option for fast text generation tasks. The technical report can be found on arXiv with the identifier 2608.00146.
Key facts
- DiffusionGemma is an experimental open-weight language model using discrete diffusion.
- It generates text by iteratively refining blocks of 256 tokens in parallel.
- It avoids the sequential decoding bottleneck of conventional autoregressive models.
- It is fine-tuned from the mixture-of-experts Gemma 4 model.
- Gemma 4 has 3.8B activated and 25.2B total parameters.
- The training pipeline uses fewer than 10% of the starting AR model's total training token budget.
- The two-stage training includes supervised fine-tuning and reinforcement learning with sampler distillation.
- DiffusionGemma establishes a new Pareto frontier for the trade-off between generation speed and quality.
Entities
Institutions
- arXiv