Probing Spatial Concepts in LLMs: Abstraction, Compositionality, and Grounding

A recent paper published on arXiv (2608.07353) presents a benchmark centered on concepts to evaluate Large Language Models (LLMs) regarding their grasp of spatial ideas. This research emphasizes fundamental characteristics of concepts: abstraction, compositionality, and groundness. The benchmark examines spatial notions like direction, distance, and topology, employing question-answering tasks as a measure. Comprehensive experiments were carried out across various LLM architectures and training approaches to assess the influence of model scale and design on conceptual comprehension. Findings highlight the shortcomings of LLMs in true concept understanding, despite their strong performance across diverse tasks. Authored by researchers, the paper is accessible as a preprint on arXiv, aiming to refine control over core concepts and their attributes.