Adaptive Steering for Discrete Diffusion Language Models
A new study from arXiv (2605.10971) reveals that uniform intervention across denoising steps degrades quality in discrete diffusion language models (DLMs), especially when steering multiple attributes. Using sparse autoencoders on four DLMs (124M to 8B parameters), researchers found that attributes commit on distinct schedules: topic within the first 2% of denoising, sentiment over 20%. They propose an adaptive scheduler that concentrates interventions on critical steps, improving controlled generation. The paper is authored by researchers from an undisclosed institution and was published on May 10, 2025.
Key facts
- Discrete diffusion language models generate text by iteratively denoising all positions in parallel.
- Uniform intervention at every denoising step degrades quality.
- Damage compounds when multiple attributes are steered jointly.
- Sparse autoencoders were trained on four DLMs (124M-8B parameters).
- Topic commits within the first 2% of denoising.
- Sentiment emerges gradually over 20% of the process.
- An adaptive scheduler concentrates interventions on critical steps.
- The paper is available on arXiv with ID 2605.10971.
Entities
—