FP4 Training Stabilized via 2D Block Quantization
A new method for training large language models with 4-bit floating point (FP4) precision addresses instability caused by scale inconsistency in tensor transposition. Conventional 1D block quantization assigns different scaling factors to the same values during forward and backward passes, leading to biased gradient updates. The proposed framework uses 2D block FP4 quantization to enforce transposition-invariant scaling, preserving consistency. Combined with truncation-free scaling and stochastic rounding, it controls quantization error and maintains unbiased gradients. The technique aims to improve training efficiency by reducing precision from FP8 to FP4 without sacrificing stability.
Key facts
- FP4 training faces instability due to scale inconsistency from tensor transposition.
- 1D block quantization assigns different scaling factors to the same values after transposition.
- The proposed method uses 2D block FP4 quantization for transposition-invariant scaling.
- Truncation-free scaling and stochastic rounding are used to control quantization error.
- The goal is to improve LLM training efficiency by reducing precision to FP4.
- The paper is from arXiv:2607.24953v1.
- The approach preserves consistency between forward and backward computations.
- Unbiased gradients are maintained through the proposed techniques.
Entities
Institutions
- arXiv