DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling

Researchers have introduced DiLaDiff, a new variant of masked diffusion language models that tackles the balance between sampling quality and throughput by creating a continuous latent space with semantic functions. This model consists of three key elements: an auto-encoder that is fine-tuned from a pre-existing masked diffusion language model, a latent diffusion model that learns the prior of the encoder distribution, and a consistency model that distills this learned prior into a streamlined latent generative model. Findings indicate that the latent-guided diffusion model surpasses the masked diffusion baseline in performance, while also speeding up inference. Additionally, consistency distillation significantly lowers computational demands, rendering latent generation almost trivial compared to discrete decoding.