Infilling Extraction Reveals Higher Memorization in Diffusion Language Models

A new study has introduced a technique called 'infilling extraction,' which aims to evaluate how diffusion language models (DLMs) retain training data. The findings show that DLMs can reveal up to three times more exact sequences than previously thought. Unlike autoregressive models, DLMs can denoise masked tokens from any location, making prefix-only probing ineffective. This approach uses a binary mask to create a bidirectional inductive bias. When tested on LLaDA-8B and Dream-7B in five different extraction modes, three training pipelines, and three datasets, it was found that edge-conditioned masks provided the best data extraction results. You can find the complete paper on arXiv.