RRQ: Single-Checkpoint Multi-Precision Quantization for LLMs

A novel post-training quantization approach called Recurrent Residual Quantization (RRQ) facilitates various effective precisions from a single model checkpoint. This innovation meets the demand for adaptable trade-offs among accuracy, memory usage, and throughput when deploying large language models (LLMs) in various environments. Unlike traditional techniques that necessitate distinct checkpoints for each desired bit-width, RRQ encodes weights as a low-bit quantized base complemented by a series of quantized residual adjustments. Initiating from a 2-bit model derived through post-training quantization (PTQ) or round-to-nearest (RTN), RRQ incrementally incorporates lightweight 2-bit residuals via RTN to create 4-, 6-, and 8-bit versions. This method does not require calibration and bypasses joint multi-bit optimization. In the Qwen3-8B configuration, the complete all-RTN 2-/4-/6-/8-bit package is assembled in 1,293 seconds, which is 3.3 times quicker than the baseline. The research can be found on arXiv under ID 2608.04048.