LLM Reliability Varies Significantly Under Meaning-Preserving Paraphrases

A recent study published on arXiv (2607.22554) examines the reactions of large language models (LLMs) to paraphrased questions that maintain their meaning. Researchers analyzed 13 models across four benchmarks and discovered that, although the overall accuracy remains relatively stable, the behavior at the instance level is inconsistent. Models often switch between right and wrong responses based on how questions are phrased, with mismatch rates surpassing 23%. Furthermore, when focusing on initially correct answers, the failure rates are even more pronounced, suggesting that the accuracy of a single prompt is not a reliable measure of performance.