BenchDrift: LLM Performance Varies with Problem Phrasing

A new study from arXiv (2608.11694) introduces BenchDrift, a tool that quantifies how rephrasing benchmark problems affects large language model (LLM) performance. The research shows that a single phrasing of a problem does not represent the entire space of possible phrasings, and that meaning-preserving variations can flip a model's answer in both directions—turning failures into successes and successes into failures. The study tests eight models across three benchmarks (GSM8K, MMLU, MATH-Hard) and finds that drift is substantial. Notably, phrasing sensitivity does not diminish as models improve; instead, it changes sign. Weaker models tend to benefit from rephrasing, while stronger models lose more than they gain. The tool generates variations along four axes: linguistic, referential, pragmatic, and structural. The findings highlight the fragility of benchmark scores and suggest that current evaluation methods may be misleading.