PreDiff-LM: Hybrid Attention Improves Discrete Masked Diffusion Language Models

A recent study presents PreDiff-LM, a technique designed to modify pretrained autoregressive transformers for discrete masked diffusion language modeling. The primary advancement lies in a unique hybrid attention mask that maintains causal attention in the visible prompt while enabling complete bidirectional attention in the masked target. When evaluated against a matched GPT-2 Medium setup on WikiText-103 with 90K steps, this hybrid approach enhances unconditional perplexity from 34.1 to 28.7 and MAUVE scores from 0.71 to 0.78 compared to uniform bidirectional attention with identical AR initialization. Additionally, attention adaptation works in conjunction with a DiffuGPT-style objective adaptation, achieving a perplexity of 26.9. Pretrained initialization significantly cuts down the steps needed to achieve a perplexity below 50 from approximately 350K to 8K, although a compute-matched fine-tuned AR model still performs well.