Collapse-and-Refine Mechanism Explains Diffusion Model Learning on Manifolds
A recent theoretical study published on arXiv (2605.20235) reveals a collapse-and-refine mechanism in diffusion models applicable to data situated on low-dimensional manifolds. When noise levels are low, the singularity of the score function leads to a swift dimensional collapse of the denoising map onto the data manifold. At moderate noise levels, the training process enhances the intrinsic density on the learned manifold. The authors introduce Score-induced Latent Diffusion (SiLD), a two-phase framework that integrates both manifold learning and density estimation through a unified denoising score matching objective, eliminating the KL regularization used in VAE-based latent diffusion models. The study demonstrates that sample complexity is contingent on intrinsic dimension, avoiding the curse of dimensionality.
Key facts
- Paper arXiv:2605.20235 identifies collapse-and-refine mechanism in diffusion models
- Mechanism driven by geometry of score function at different noise scales
- Small noise scales: diverging singularity causes dimensional collapse onto data manifold
- Moderate noise scales: training refines intrinsic density on learned manifold
- Proposes Score-induced Latent Diffusion (SiLD) framework
- SiLD replaces KL regularization of VAE-based latent diffusion models
- Sample complexity depends on intrinsic dimension, not ambient dimension
- Bypasses curse of dimensionality for data on low-dimensional manifolds
Entities
Institutions
- arXiv