UCCL-Zip: Lossless Compression for GPU Communication in LLMs

The UCCL-Zip system introduces lossless compression into GPU communication primitives to tackle the limitations posed by large language models. In contrast to earlier techniques that relied on quantization or lossy compression, UCCL-Zip maintains numerical accuracy. It facilitates both point-to-point (P2P) and collective communication without altering user APIs. For P2P, Uzip-P2P employs a split-send pipeline that allows compression to occur simultaneously with communication. In the case of collective communication, Uzip-NCCL incorporates compression into NCCL's persistent kernel model through fused execution, which minimizes unnecessary memory traffic and kernel launches.