MDLMPE: New Positional Encoding for Masked Diffusion Language Models

A recent study published on arXiv (ID: 2608.03769) presents MDLMPE, a novel positional encoding technique tailored for masked diffusion language models (MDLMs). In contrast to autoregressive (AR) models, which reveal a continuous prefix during decoding, MDLM denoising generates varying, non-contiguous arrangements of visible and hidden tokens. Traditional positional encodings, such as RoPE, effectively capture sequence order and pairwise differences but fail to adapt to the shifting token availability. MDLMPE stands out as the first approach to incorporate awareness of this evolving configuration in positional representations. It utilizes a binary sequence to denote token availability, employs distance-aware Gaussian weighting, and transforms the resulting pattern via a cosine basis for a distribution-aware positional representation. The full paper can be accessed at https://arxiv.org/abs/2608.03769.