RoundPipe: Near-Zero-Bubble Pipeline for LLM Training on Consumer GPUs

A new pipeline schedule called RoundPipe addresses the weight binding issue in fine-tuning Large Language Models (LLMs) on consumer-grade GPUs. By treating GPUs as stateless execution workers and dispatching computation stages in a round-robin manner, RoundPipe achieves near-zero pipeline bubbles. This method breaks the constraint of binding uneven model stages to specific GPUs, which previously limited throughput. The approach integrates with CPU offloading to mitigate memory and PCIe interconnect bottlenecks, making LLM fine-tuning more efficient on multiple consumer GPUs.