Quantization Degradation in LLMs: A Signal-Noise Perspective

A recent paper on arXiv (2608.08188) explores the impact of post-training quantization on large language models (LLMs), indicating that performance decline is not exclusively linked to bit-width. The research methodically analyzes weight-only quantization across various model families, differing bit-widths, quantization techniques, model sizes, and downstream tasks. Significant results indicate that 4-bit quantization generally maintains performance, whereas 2-bit often leads to extensive degradation. At 3-bit, performance decline is noticeable but varies significantly depending on the task type, quantization approach, and model size. To clarify this variability, the authors propose a signal-to-noise ratio (SNR) framework to assess how quantization disrupts full-precision representations. They identify two interconnected processes responsible for degradation: the creation of quantization errors in individual modules and their accumulation through layers. An SNR decomposition shows that newly introduced errors are the main factors, while error accumulation across layers follows a consistent pattern. This paper lays a theoretical foundation for understanding why different configurations produce varied results, providing valuable insights for enhancing quantization strategies in LLM applications.