Self-Conditioned Masked Diffusion Models Improve Discrete Sequence Generation
A new approach called Self-Conditioned Masked Diffusion Models (SCMDM) has been released on arXiv. Unlike traditional masked diffusion models (MDMs) that create sequences through repeated denoising, SCMDM addresses a key issue: when a token is still masked after a reverse update, the model misses the chance to predict its clean state. This can lead to problems in refining outputs across steps. SCMDM solves this by using the model's previous clean-state predictions for each denoising step, which requires only slight adjustments to the existing framework. This method eliminates the need for complicated pathways or extra evaluations during sampling, representing a major improvement over older, more resource-heavy self-conditioning techniques. You can check it out on arXiv with ID 2604.26985v1.
Key facts
- Method called Self-Conditioned Masked Diffusion Models (SCMDM)
- Proposed as post-training adaptation for masked diffusion models (MDMs)
- Addresses limitation where still-masked positions are inferred from mask token alone
- Conditions each denoising step on model's own previous clean-state predictions
- Requires minimal architectural change
- No recurrent latent-state pathway introduced
- No auxiliary reference model needed
- No extra denoiser evaluations during sampling
Entities
Institutions
- arXiv