Masked Diffusion Models Prioritize Entities in Graph-to-Text Generation
A recent study available on arXiv (2605.31564) marks the inaugural comprehensive examination of masked diffusion language models (MDLMs) in the context of graph-to-text generation. The researchers discovered that MDLMs reveal tokens in a unique sequence: starting with entities, followed by relational and functional words, and concluding with structural tokens. This approach differs from the linear generation seen in autoregressive LLMs. Additionally, the study uncovered an unreported failure mode in supervised fine-tuning (SFT), which disrupts the token unmasking strategy by prematurely fixing structural sentence-ending tokens, leading to output length issues and potential omissions or hallucinations. To address this, the authors suggest lambda-scaled structural decoding, a modification that downweights structural token confidence, resulting in a +9.4 BLEU-4 score improvement. Furthermore, the study introduces Graph-LLaDA, which combines a graph encoder with LLaDA for improved performance.
Key facts
- First systematic study of masked diffusion language models for graph-to-text generation.
- MDLMs unmask entities first, then relational/function words, structural tokens last.
- SFT disrupts this strategy by anchoring structural tokens early, fixing output length.
- Lambda-scaled structural decoding recovers +9.4 BLEU-4.
- Graph-LLaDA integrates a graph encoder with LLaDA.
- Study published on arXiv with ID 2605.31564.
- Contrasts with autoregressive LLMs' linear generation.
- Training-free inference-time modification proposed.
Entities
Institutions
- arXiv