IntentGrasp Benchmark Reveals LLMs Struggle with Intent Understanding

IntentGrasp, a newly established benchmark, assesses how well large language models (LLMs) comprehend intent. It is based on 49 open-licensed datasets spanning 12 different domains, featuring a training set comprising 262,759 instances. Additionally, there are two evaluation sets: the All Set, which contains 12,909 test cases, and the more difficult Gem Set, with 470 cases. Evaluations conducted on 20 LLMs from 7 different families, including advanced models such as GPT-5.4, Gemini-3.1-Pro, and Claude-Opus-4.7, revealed that scores fell below 60% on the All Set and under 25% on the Gem Set. Alarmingly, 17 out of the 20 models performed worse than random guessing on the Gem Set, indicating substantial deficiencies in current LLM capabilities.