Multi-Block Editing Improves Discrete Diffusion Language Models
To tackle the block boundary issue in block diffusion language models, researchers have introduced Multi-Block Editing (MBE). Block diffusion is the leading approach for enhancing discrete diffusion language models (dLLMs), allowing for text decoding in fixed-size segments, which facilitates parallel generation and manageable attention costs. Nevertheless, tokens positioned at the edges of blocks miss out on future cross-block context, and completed blocks create irreversible context that can lead to uncertainty. MBE presents a decoding algorithm that requires no training, enabling the modification of previously decoded tokens by utilizing cross-block context to reduce error accumulation. This technique has been validated in arXiv:2607.22663.
Key facts
- Block diffusion is the dominant paradigm for scaling discrete diffusion language models (dLLMs).
- Decoding text in fixed-size blocks preserves parallel generation within each block.
- Quadratic attention cost remains tractable with block diffusion.
- Tokens near block ends are generated without access to future cross-block context.
- Once a block is finalized, its uncertain predictions become irreversible context for subsequent blocks.
- This creates a block boundary problem where uncertainty accumulates toward block boundaries.
- Multi-Block Editing (MBE) mitigates this problem by editing decoded tokens based on cross-block context.
- MBE proposes a training-free decoding algorithm to edit tokens in previous blocks.
Entities
Institutions
- arXiv