Neural Program Synthesis Generalization Mapped via Arithmetic Grammar

A new arXiv preprint (2604.27551) introduces a controlled environment using a domain-specific arithmetic grammar to evaluate generalization in large-scale transformers for program synthesis. By enumerating millions of unique programs and constructing interpretable syntactic and semantic metric spaces, researchers precisely map data distributions and isolate distributional shifts. Experiments show that optimizing density generalization through diverse sampling induces robust out-of-distribution generalization, while support generalization reveals severe struggles for transformers. The study aims to distinguish true generalization from memorized templates, addressing data contamination and opaque training corpora.