MegaSlide-DiT: Memory-Efficient Video Diffusion on Single GPU
Researchers have introduced MegaSlide-DiT, a prototype that adapts a pre-trained 105-billion-parameter Diffusion Transformer (DiT) for high-resolution video generation on a single NVIDIA H200 GPU with 1.5 TB of host RAM. The system addresses two major memory bottlenecks: parameter memory and activation memory. By keeping all persistent weights, master weights, and optimizer moments in host memory and streaming only transient shards to the GPU on demand, MegaSlide-DiT eliminates the need for large GPU clusters. It also replaces quadratic global self-attention with 3D deformable local attention to reduce activation memory. The approach is detailed in a paper published on arXiv (2607.22696).
Key facts
- MegaSlide-DiT adapts a 105-billion-parameter DiT on a single H200 GPU.
- System uses 1.5 TB of host RAM for persistent model state.
- Transient shards are streamed to GPU on demand.
- Replaces quadratic global attention with 3D deformable local attention.
- Addresses parameter memory and activation memory walls.
- Paper published on arXiv with ID 2607.22696.
- Built on Diffusion Transformers (DiTs) for high-resolution video.
- Designed to avoid large GPU clusters for adaptation.
Entities
Institutions
- arXiv