Unsloth and NVIDIA Collaborate to Accelerate LLM Training by ~25%

Unsloth has partnered with NVIDIA to introduce optimizations that make large language model (LLM) training approximately 25% faster without any loss in accuracy. These improvements build on Unsloth's existing 2-5x speedup and are automatically enabled on RTX laptops, data center GPUs, and DGX Spark machines upon updating Unsloth. The collaboration addresses three key areas: packed-sequence caching, double-buffered activation checkpointing, and efficient Mixture-of-Experts (MoE) routing. Packed-sequence caching reduces overhead by reusing metadata across transformer layers instead of reconstructing it each time. Double-buffered activation checkpointing overlaps data transfer with computation, hiding latency. MoE routing optimizations minimize dynamic indexing queries. Benchmarks on NVIDIA B200 Blackwell GPUs show consistent speedups across larger dense models with minimal extra VRAM usage. The final losses remain effectively unchanged, confirming that these optimizations preserve model quality.