Data, Not Architecture, Drives Word Order Preferences in Language Models

A recent investigation published on arXiv (2608.15129) thoroughly examines the preferences for word order in decoder-only language models, analyzing 192 artificial languages alongside a variety of natural languages. The findings indicate that these models favor left-branching structures in artificial languages, which do not correspond with established natural language universals or human biases in word order acquisition. For natural languages, monolingual models initially show no distinct base word order preference at smaller data scales; however, as the dataset increases, a tendency towards right-branching subject-verb-object (SVO) structures becomes apparent, while subject-object-verb (SOV) structures, despite being the most common globally, lag behind. This SVO preference is also evident in multilingual models, linked more to the quality and quantity of language resources than to specific word orders. The research, conducted by teams from MIT, Stanford, and Google Research, was released on August 26, 2025, and it questions the notion of inherent architectural biases in language models, emphasizing the significant impact of training data composition on syntactic preferences.