Transformers' Length Generalization Characterized for Regular Languages

A recent theoretical study published on arXiv (2608.13433) delivers the first comprehensive description of the regular languages for which transformer-based language models can generalize in length. It introduces a polynomial-time decision algorithm that depends on the size of the syntactic monoid of the language. This research fills a crucial gap, as it was previously unclear which tasks allowed for such generalization, even among regular languages. The authors successfully characterize these languages using C-RASP, a formalism that identifies the languages transformers can length-generalize on. Traditional methods like Krohn-Rhodes decomposition are inadequate for C-RASP, necessitating new algebraic techniques specifically designed for it. This work is vital for the AI field, as it establishes a solid framework for predicting length generalization in transformers, influencing model design for tasks with varying input lengths.