MixQuant: Adaptive Mixed-Precision Quantization for LLMs
MixQuant, a novel framework, tackles the issue of adaptive mixed-precision quantization for large language models (LLMs). Traditional mixed-precision techniques assign higher bitwidths to critical layers but are limited to a single fixed memory budget, which can differ across implementations and remains unknown during calibration. Adaptive quantization seeks to accommodate any budget with a single offline calibration; however, existing scoring methods overlook how a layer's sensitivity relates to the quantization levels of surrounding layers. This technique-agnostic adaptive framework can integrate with any base quantizer. It evaluates each layer's distortion across various random quantized upstream configurations to generate budget-agnostic scores, subsequently calibrating the quantizer's parameters based on those scores. The research highlights that a layer's sensitivity is significantly influenced by the bitwidths of its upstream layers, altering the optimal bit allocation. The paper is available on arXiv under ID 2607.23047.
Key facts
- MixQuant is a technique-agnostic adaptive framework for mixed-precision quantization of LLMs.
- It addresses the problem of varying memory budgets across deployments.
- Existing methods solve allocation for a single fixed budget.
- Current adaptive methods score layer sensitivity without considering dependency on other layers' quantization levels.
- MixQuant marginalizes each layer's distortion over random quantized upstream configurations.
- It obtains budget-agnostic scores and calibrates quantizer parameters on plans.
- The paper demonstrates that a layer's sensitivity depends on upstream bitwidths.
- The work is published on arXiv with ID 2607.23047.
Entities
Institutions
- arXiv