Statistical Framework Detects LLM Degradation

A new statistical approach uses McNemar's test to detect degradation in large language models (LLMs) even after theoretically lossless optimizations. The method compares model outputs on paired samples to identify significant accuracy drops while controlling false positives. This addresses the problem that numerical errors can cause non-robust generations at temperature zero. The framework is proposed as a tool for ensuring model quality during inference cost and latency reduction efforts.