LLMs Struggle with Context-Free Grammar Interpretation

A recent investigation published on arXiv (2604.20811) assesses large language models as interpreters of new context-free grammars in context. The team presents RoboGrid, a framework designed to evaluate LLMs on various aspects such as syntax, behavior, and semantics via rigorous stress-tests focusing on recursion depth, expression complexity, and surface styles. Findings indicate a hierarchical decline: while LLMs retain surface syntax, they struggle to maintain structural semantics. Although chain-of-thought reasoning provides some improvement, performance significantly deteriorates with deep recursion and extensive branching, leading to a loss of semantic alignment at extreme depths. Additionally, the employment of 'Alien' lexicons highlights a dependency on semantic bootstrapping from keywords instead of pure symbol processing.